[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:19 dist-test-slave-b5p5 krb5kdc[28093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:19 dist-test-slave-b5p5 krb5kdc[28093](info): set up 2 sockets
Sep 28 02:43:19 dist-test-slave-b5p5 krb5kdc[28093](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:43:20.186975 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:20.300256 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45287
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:45287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:20.378700 28106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:20.378881 28106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:20.378902 28106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:20.380021 28106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:20.380061 28106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:20.380075 28106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:20.380088 28106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:20.380100 28106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:20.381525 28106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:45287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:20.381762 28106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:20.381987 28106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:20.386511 28114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.386546 28111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.386524 28112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:20.386878 28106 server_base.cc:1034] running on GCE node
I20260928 02:43:20.387108 28106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:20.387392 28106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:20.388556 28106 hybrid_clock.cc:648] HybridClock initialized: now 1790563400388540 us; error 48 us; skew 500 ppm
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:20.391037 28106 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:20.391396 28106 webserver.cc:460] Webserver started at http://127.27.109.190:38475/ using document root <none> and password file <none>
I20260928 02:43:20.391580 28106 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:20.391619 28106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:20.391707 28106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:20.392624 28106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "2572adbc9ccd4589baed2f2db0f22981"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "6bb3300dc9225b843dd9fc1e78e0e51c"
server_key_iv: "326a23cc854caa864af8dc012e54d510"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.392910 28106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "2572adbc9ccd4589baed2f2db0f22981"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "6bb3300dc9225b843dd9fc1e78e0e51c"
server_key_iv: "326a23cc854caa864af8dc012e54d510"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.394089 28106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:20.394878 28121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.395052 28106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.395136 28106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "2572adbc9ccd4589baed2f2db0f22981"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "6bb3300dc9225b843dd9fc1e78e0e51c"
server_key_iv: "326a23cc854caa864af8dc012e54d510"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.395210 28106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:20.418828 28106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:20.420612 28106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:20.420753 28106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:20.424839 28106 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:45287
I20260928 02:43:20.424885 28173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:45287 every 8 connection(s)
I20260928 02:43:20.425235 28106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:20.426082 28174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:20.428393 28174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981: Bootstrap starting.
I20260928 02:43:20.428921 28174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:20.429138 28174 log.cc:824] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:20.429867 28174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981: No bootstrap required, opened a new log
I20260928 02:43:20.431427 28174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } }
I20260928 02:43:20.431556 28174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:20.431579 28174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2572adbc9ccd4589baed2f2db0f22981, State: Initialized, Role: FOLLOWER
I20260928 02:43:20.431717 28174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } }
I20260928 02:43:20.431790 28174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:20.431811 28174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:20.431838 28174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:20.432292 28174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } }
I20260928 02:43:20.432374 28174 leader_election.cc:302] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2572adbc9ccd4589baed2f2db0f22981; no voters:
I20260928 02:43:20.432490 28174 leader_election.cc:288] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:20.432551 28177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:20.432741 28177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [term 1 LEADER]: Becoming Leader. State: Replica: 2572adbc9ccd4589baed2f2db0f22981, State: Running, Role: LEADER
I20260928 02:43:20.432762 28174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:20.432828 28177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } }
I20260928 02:43:20.433197 28179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2572adbc9ccd4589baed2f2db0f22981. Latest consensus state: current_term: 1 leader_uuid: "2572adbc9ccd4589baed2f2db0f22981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } } }
I20260928 02:43:20.433321 28178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2572adbc9ccd4589baed2f2db0f22981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2572adbc9ccd4589baed2f2db0f22981" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45287 } } }
I20260928 02:43:20.433346 28179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:20.433411 28178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:20.433772 28186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:20.434195 28186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:20.435169 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28106
I20260928 02:43:20.435297 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:20.435883 28086 external_mini_cluster.cc:1428] Setting key 41991a27e30871ae17f3d63452cacf36
I20260928 02:43:20.436775 28186 catalog_manager.cc:1347] Generated new cluster ID: f19c7a66f56348e987e25e97631a1966
I20260928 02:43:20.436825 28186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:20.452242 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.437057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.27.109.190:45287)
0928 02:43:20.437186 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:20.437194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:20.437225 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:43:20.439176 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:20.439183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:20.439237 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:20.439374 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:20.441151 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.442563 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.443871 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.444437 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.444579 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.448494 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:20.448498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:20.448498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:20.448516 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:20.450348 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.450824 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.450827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.450827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.450921 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.451191 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.451191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.451192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.451484 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:20.451627 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:20.451920 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:43:20.452042 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260928 02:43:20.463210 28186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:20.463876 28186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:20.472157 28186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2572adbc9ccd4589baed2f2db0f22981: Generated new TSK 0
I20260928 02:43:20.472303 28186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:20.503640 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:20.582442 28200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:20.582696 28200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:20.582726 28200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:20.583863 28200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:20.583904 28200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:20.583931 28200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:20.585377 28200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:20.585628 28200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:20.585855 28200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:20.590598 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:20.590706 28200 server_base.cc:1034] running on GCE node
W20260928 02:43:20.590622 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.590826 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:20.591030 28200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:20.591274 28200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:20.592432 28200 hybrid_clock.cc:648] HybridClock initialized: now 1790563400592429 us; error 35 us; skew 500 ppm
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:20.594728 28200 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:20.595116 28200 webserver.cc:460] Webserver started at http://127.27.109.129:35511/ using document root <none> and password file <none>
I20260928 02:43:20.595341 28200 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:20.595410 28200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:20.595552 28200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:20.596506 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "b199eb1937c540bf84afd15e4fe12f10"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "39b178a0ef2e0c64ffea1f87249c6234"
server_key_iv: "cdbad23998def0137b1a355fd419f184"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.596848 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "b199eb1937c540bf84afd15e4fe12f10"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "39b178a0ef2e0c64ffea1f87249c6234"
server_key_iv: "cdbad23998def0137b1a355fd419f184"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.598213 28200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:20.598949 28215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.599105 28200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.599200 28200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "b199eb1937c540bf84afd15e4fe12f10"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "39b178a0ef2e0c64ffea1f87249c6234"
server_key_iv: "cdbad23998def0137b1a355fd419f184"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.599278 28200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:20.608184 28200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:20.609788 28200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:20.609934 28200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:20.610132 28200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:20.610405 28200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:20.610456 28200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.610503 28200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:20.610548 28200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.615929 28200 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37853
I20260928 02:43:20.616017 28328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37853 every 8 connection(s)
I20260928 02:43:20.616289 28200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:20.617779 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28200
I20260928 02:43:20.617892 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:20.618065 28086 external_mini_cluster.cc:1428] Setting key 139b528ac504264ed5c035ad0eb6481e
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
I20260928 02:43:20.626103 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.616918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:49275 (local address 127.27.109.190:45287)
0928 02:43:20.616957 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:20.616959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:20.617125 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:43:20.617564 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:20.617565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:20.617576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:20.617644 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:20.618819 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.619305 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.620557 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.620899 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.620972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.623270 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:20.623271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:20.623272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:20.623279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:20.624731 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.625140 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.625141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.625141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.625203 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.625460 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.625461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.625462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.625574 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:20.625643 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:20.625932 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:43:20.626015 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:20.626278 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.616880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45287 (local address 127.27.109.129:49275)
0928 02:43:20.617022 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:20.617058 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:20.617489 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:20.617761 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:20.617765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:20.618442 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:20.618730 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.618734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.619392 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.619393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.620454 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.620456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.620988 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.620989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.621119 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.621559 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:20.621569 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:20.623161 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:20.624832 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.624839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.624841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.625061 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.625272 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.625273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.625273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.625384 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.625646 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:20.625652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:20.625804 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:43:20.625910 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:43:20.626052 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 02:43:20.626780 28329 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45287
I20260928 02:43:20.626870 28329 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:20.627031 28329 heartbeater.cc:511] Master 127.27.109.190:45287 requested a full tablet report, sending...
I20260928 02:43:20.627528 28138 ts_manager.cc:194] Registered new tserver with Master: b199eb1937c540bf84afd15e4fe12f10 (127.27.109.129:37853)
I20260928 02:43:20.628106 28138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:49275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:20.641309 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:20.719411 28334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:20.719600 28334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:20.719623 28334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:20.720679 28334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:20.720714 28334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:20.720741 28334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:20.722044 28334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:20.722203 28334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:20.722355 28334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:20.727095 28339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.727082 28340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.727084 28342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:20.727350 28334 server_base.cc:1034] running on GCE node
I20260928 02:43:20.727562 28334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:20.727788 28334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:20.728947 28334 hybrid_clock.cc:648] HybridClock initialized: now 1790563400728933 us; error 26 us; skew 500 ppm
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:20.731407 28334 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:20.731765 28334 webserver.cc:460] Webserver started at http://127.27.109.130:45363/ using document root <none> and password file <none>
I20260928 02:43:20.731927 28334 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:20.731966 28334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:20.732048 28334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:20.733078 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "eb19f1202881481fa46ee7c8f73aac72"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "b90612e40ff120e73422119646f4d997"
server_key_iv: "86f6da0b4bcf257cb34399ef89c6f770"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.733486 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "eb19f1202881481fa46ee7c8f73aac72"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "b90612e40ff120e73422119646f4d997"
server_key_iv: "86f6da0b4bcf257cb34399ef89c6f770"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.734786 28334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:20.735395 28349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.735549 28334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:20.735615 28334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "eb19f1202881481fa46ee7c8f73aac72"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "b90612e40ff120e73422119646f4d997"
server_key_iv: "86f6da0b4bcf257cb34399ef89c6f770"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.735664 28334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:20.753129 28334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:20.755407 28334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:20.755517 28334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:20.755690 28334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:20.755932 28334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:20.755962 28334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.755982 28334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:20.755995 28334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.761693 28334 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44621
I20260928 02:43:20.761771 28462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44621 every 8 connection(s)
I20260928 02:43:20.761983 28334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:20.765525 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28334
I20260928 02:43:20.765652 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:20.765838 28086 external_mini_cluster.cc:1428] Setting key 932c38ce25db0acd1e083bbc6cdef3bd
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:20.771657 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.762659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45287 (local address 127.27.109.130:39585)
0928 02:43:20.762809 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:20.762847 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:20.763323 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:20.763523 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:20.763527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:20.764051 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:20.764238 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.764242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.765036 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.765036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.765863 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.765865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.766255 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.766256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.766380 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.766925 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:20.766934 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:20.768506 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:20.770299 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.770305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.770308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.770609 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.770823 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.770823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.770824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.770917 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.771204 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:20.771208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:20.771313 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:43:20.771397 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:43:20.771508 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 02:43:20.771673 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.762687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:39585 (local address 127.27.109.190:45287)
0928 02:43:20.762749 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:20.762751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:20.762933 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:43:20.763419 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:20.763420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:20.763430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:20.763472 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:20.764341 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.764940 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.765954 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.766168 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.766248 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.768599 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:20.768600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:20.768600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:20.768609 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:20.770193 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.770682 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.770683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.770683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.770750 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.770982 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.770983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.770984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.771129 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:20.771211 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:20.771447 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:43:20.771537 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:43:20.772143 28463 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45287
I20260928 02:43:20.772249 28463 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:20.772414 28463 heartbeater.cc:511] Master 127.27.109.190:45287 requested a full tablet report, sending...
I20260928 02:43:20.772837 28138 ts_manager.cc:194] Registered new tserver with Master: eb19f1202881481fa46ee7c8f73aac72 (127.27.109.130:44621)
I20260928 02:43:20.773214 28138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:39585
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:20.793110 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:45059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:20.871402 28468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:20.871585 28468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:20.871608 28468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:20.872741 28468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:20.872785 28468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:20.872812 28468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:20.874217 28468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:45059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:20.874482 28468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:20.874738 28468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:20.879590 28476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.879719 28474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:20.879598 28473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:20.879824 28468 server_base.cc:1034] running on GCE node
I20260928 02:43:20.880012 28468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:20.880190 28468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:20.881322 28468 hybrid_clock.cc:648] HybridClock initialized: now 1790563400881315 us; error 33 us; skew 500 ppm
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:20.883687 28468 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:20.884133 28468 webserver.cc:460] Webserver started at http://127.27.109.131:39491/ using document root <none> and password file <none>
I20260928 02:43:20.884356 28468 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:20.884423 28468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:20.884580 28468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:20.885583 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "4c1151d0b325482eb0cd4d7c69f95fa9"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "a347c4e56625143dd9cf32df3790cd52"
server_key_iv: "55d53e02bfe397c711e531d55b7ac365"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.885933 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "4c1151d0b325482eb0cd4d7c69f95fa9"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "a347c4e56625143dd9cf32df3790cd52"
server_key_iv: "55d53e02bfe397c711e531d55b7ac365"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.887274 28468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:20.887945 28483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.888123 28468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:20.888218 28468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "4c1151d0b325482eb0cd4d7c69f95fa9"
format_stamp: "Formatted at 2026-09-28 02:43:20 on dist-test-slave-b5p5"
server_key: "a347c4e56625143dd9cf32df3790cd52"
server_key_iv: "55d53e02bfe397c711e531d55b7ac365"
server_key_version: "encryptionkey@0"
I20260928 02:43:20.888299 28468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:20.943756 28468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:20.945588 28468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:20.945757 28468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:20.945990 28468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:20.946296 28468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:20.946352 28468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.946393 28468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:20.946422 28468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:20.952216 28468 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:40919
I20260928 02:43:20.952272 28596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:40919 every 8 connection(s)
I20260928 02:43:20.952631 28468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:20.957767 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28468
I20260928 02:43:20.957860 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:20.958016 28086 external_mini_cluster.cc:1428] Setting key 896deecf4c0f3e17f3e518f51dbae778
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:20.961391 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.953360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:55007 (local address 127.27.109.190:45287)
0928 02:43:20.953425 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:20.953427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:20.953632 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:43:20.954093 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:20.954094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:20.954103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:20.954161 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:20.955104 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.955582 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.956501 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.956674 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.956721 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.958958 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:20.958960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:20.958960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:20.958967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:20.960165 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.960660 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.960661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.960661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.960712 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.960934 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.960934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.960935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.961022 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:20.961067 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:20.961277 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:43:20.961331 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:20.961614 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.953344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45287 (local address 127.27.109.131:55007)
0928 02:43:20.953509 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:20.953549 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:20.954015 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:20.954216 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:20.954219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:20.954844 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:20.955027 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.955032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.955659 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.955660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.956429 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:20.956431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.956741 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.956742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:20.956846 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.957268 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:20.957275 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:20.958868 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:20.960275 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.960281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.960283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.960595 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.960783 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:20.960783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:20.960783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:20.960885 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:20.961072 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:20.961076 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:20.961187 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:43:20.961265 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:43:20.961371 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260928 02:43:20.962072 28597 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45287
I20260928 02:43:20.962142 28597 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:20.962320 28597 heartbeater.cc:511] Master 127.27.109.190:45287 requested a full tablet report, sending...
I20260928 02:43:20.962703 28138 ts_manager.cc:194] Registered new tserver with Master: 4c1151d0b325482eb0cd4d7c69f95fa9 (127.27.109.131:40919)
I20260928 02:43:20.963061 28138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:55007
I20260928 02:43:20.969897 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:20.991768 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:20.984518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.27.109.129:37853)
0928 02:43:20.984705 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:20.984710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:20.984719 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:20.984742 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:20.984743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:20.984756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:20.984820 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:20.985327 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.986007 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.986737 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:20.987032 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:20.987087 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:20.989107 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:20.989114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:20.989114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:20.989120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:20.990754 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.991041 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.991043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.991043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.991092 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:20.991243 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:20.991244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:20.991244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:20.991347 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:20.991455 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:20.991526 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:43:20.991623 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
W20260928 02:43:20.992142 28243 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59914
Sep 28 02:43:20 dist-test-slave-b5p5 krb5kdc[28093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:43:21 dist-test-slave-b5p5 krb5kdc[28093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:21.012282 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:21.006153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.27.109.129:37853)
0928 02:43:21.006228 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:21.006230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:21.006242 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:43:21.006304 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:21.006305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:21.006315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:21.006361 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:21.006915 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:21.007367 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:21.008019 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:21.008202 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:21.008249 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:21.010237 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:21.010238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:21.010238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:21.010243 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:21.011433 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:21.011738 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:21.011739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:21.011739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:21.011783 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:21.011938 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:21.011938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:21.011939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:21.012014 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:21.012096 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:21.012161 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:43:21.012222 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:43:21.012787 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28200
I20260928 02:43:21.017763 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28334
I20260928 02:43:21.021714 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28468
I20260928 02:43:21.025727 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28106
2026-09-28T02:43:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1893 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:21 dist-test-slave-b5p5 krb5kdc[28611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:21 dist-test-slave-b5p5 krb5kdc[28611](info): set up 2 sockets
Sep 28 02:43:21 dist-test-slave-b5p5 krb5kdc[28611](info): commencing operation
krb5kdc: starting...
W20260928 02:43:22.051013 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:22.160944 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36119
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:36119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:22.240803 28624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:22.241026 28624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:22.241062 28624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:22.242205 28624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:22.242270 28624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:22.242297 28624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:22.242327 28624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:22.242363 28624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:22.243861 28624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:36119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:22.244122 28624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:22.244334 28624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:22.248980 28630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:22.248986 28629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.249161 28624 server_base.cc:1034] running on GCE node
W20260928 02:43:22.248996 28632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.249473 28624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:22.249758 28624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:22.250916 28624 hybrid_clock.cc:648] HybridClock initialized: now 1790563402250907 us; error 34 us; skew 500 ppm
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:22.253275 28624 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:22.253746 28624 webserver.cc:460] Webserver started at http://127.27.109.190:37189/ using document root <none> and password file <none>
I20260928 02:43:22.253968 28624 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:22.254037 28624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:22.254182 28624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:22.255282 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "6622f7ba8f62419bb93c695252269c6c"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "982ca79408df2509dee0b4b090408cb9"
server_key_iv: "df1712d7350bdc1bf774462951a9bd7a"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.255637 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "6622f7ba8f62419bb93c695252269c6c"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "982ca79408df2509dee0b4b090408cb9"
server_key_iv: "df1712d7350bdc1bf774462951a9bd7a"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.256904 28624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:22.257659 28639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.257848 28624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:22.257946 28624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "6622f7ba8f62419bb93c695252269c6c"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "982ca79408df2509dee0b4b090408cb9"
server_key_iv: "df1712d7350bdc1bf774462951a9bd7a"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.258025 28624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:22.272298 28624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:22.274024 28624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:22.274178 28624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:22.278168 28624 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:36119
I20260928 02:43:22.278200 28691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:36119 every 8 connection(s)
I20260928 02:43:22.278607 28624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:22.279414 28692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:22.281487 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c: Bootstrap starting.
I20260928 02:43:22.282039 28692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:22.282325 28692 log.cc:824] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:22.283032 28692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c: No bootstrap required, opened a new log
I20260928 02:43:22.284540 28692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } }
I20260928 02:43:22.284667 28692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:22.284688 28692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6622f7ba8f62419bb93c695252269c6c, State: Initialized, Role: FOLLOWER
I20260928 02:43:22.284858 28692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } }
I20260928 02:43:22.284919 28692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:22.284960 28692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:22.285006 28692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:22.285355 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28624
I20260928 02:43:22.285425 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:22.285535 28692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } }
I20260928 02:43:22.285602 28086 external_mini_cluster.cc:1428] Setting key b2068dbe22f50f23f4ca9e9aba6aa693
I20260928 02:43:22.285643 28692 leader_election.cc:302] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6622f7ba8f62419bb93c695252269c6c; no voters:
I20260928 02:43:22.285822 28692 leader_election.cc:288] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:22.285851 28695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:22.286013 28695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [term 1 LEADER]: Becoming Leader. State: Replica: 6622f7ba8f62419bb93c695252269c6c, State: Running, Role: LEADER
I20260928 02:43:22.286075 28692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:22.286115 28695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } }
I20260928 02:43:22.286398 28695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6622f7ba8f62419bb93c695252269c6c. Latest consensus state: current_term: 1 leader_uuid: "6622f7ba8f62419bb93c695252269c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } } }
I20260928 02:43:22.286507 28695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:22.286666 28697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6622f7ba8f62419bb93c695252269c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6622f7ba8f62419bb93c695252269c6c" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36119 } } }
I20260928 02:43:22.286737 28697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:22.286847 28703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:22.287380 28703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:22.289811 28703 catalog_manager.cc:1347] Generated new cluster ID: b756ea4a913e4303b31a09b69d35a546
I20260928 02:43:22.289865 28703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:22.297458 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.286173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.27.109.190:36119)
0928 02:43:22.286353 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.286362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.286620 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:43:22.286762 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.286769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.286836 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.286963 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.287876 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.288707 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.289814 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.290454 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.290611 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.293252 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.293255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.293256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.293266 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.295856 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.296255 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.296257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.296258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.296360 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.296570 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.296570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.296571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.296848 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.296982 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.297108 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.297235 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260928 02:43:22.304517 28703 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:22.304983 28703 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:22.309337 28703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6622f7ba8f62419bb93c695252269c6c: Generated new TSK 0
I20260928 02:43:22.309510 28703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:22.345069 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:22.430963 28718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:22.431155 28718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:22.431180 28718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:22.432240 28718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:22.432271 28718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:22.432296 28718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:22.433650 28718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:22.433811 28718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:22.433969 28718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:22.438736 28726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:22.438936 28723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.439014 28718 server_base.cc:1034] running on GCE node
W20260928 02:43:22.438952 28724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.439270 28718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:22.439509 28718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:22.440657 28718 hybrid_clock.cc:648] HybridClock initialized: now 1790563402440639 us; error 37 us; skew 500 ppm
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:22.443077 28718 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:22.443487 28718 webserver.cc:460] Webserver started at http://127.27.109.129:43125/ using document root <none> and password file <none>
I20260928 02:43:22.443714 28718 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:22.443760 28718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:22.443883 28718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:22.444865 28718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "a16368da92339ed354e6f096a195610e"
server_key_iv: "5f8f005c1d96786989398da8f5588441"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.445204 28718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "a16368da92339ed354e6f096a195610e"
server_key_iv: "5f8f005c1d96786989398da8f5588441"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.446593 28718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:22.447322 28733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.447487 28718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:22.447619 28718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "a16368da92339ed354e6f096a195610e"
server_key_iv: "5f8f005c1d96786989398da8f5588441"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.447698 28718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:22.456496 28718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:22.458146 28718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:22.458299 28718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:22.458603 28718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:22.458894 28718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:22.458947 28718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.458989 28718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:22.459015 28718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.464690 28718 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37811
I20260928 02:43:22.464773 28846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37811 every 8 connection(s)
I20260928 02:43:22.465060 28718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:22.469954 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28718
I20260928 02:43:22.470058 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:22.470223 28086 external_mini_cluster.cc:1428] Setting key 8b4942f0b819b4f97eccdabc8bbf4b24
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:22.474175 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.465744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:41943 (local address 127.27.109.190:36119)
0928 02:43:22.465815 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.465816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.465995 (+ 179us) server_negotiation.cc:407] Connection header received
0928 02:43:22.466934 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.466935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.466947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.466999 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.467520 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.468000 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.468909 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.469089 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.469138 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.471384 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.471386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.471386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.471394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.472818 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.473268 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.473269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.473270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.473329 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.473570 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.473571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.473572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.473680 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.473752 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.474012 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.474088 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:43:22.474366 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.465721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36119 (local address 127.27.109.129:41943)
0928 02:43:22.465866 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.465903 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.466845 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.467047 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.467050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.467279 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:22.467447 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.467451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.468085 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.468086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.468835 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.468836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.469160 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.469161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.469252 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.469638 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:22.469646 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:22.471288 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:22.472919 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.472926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.472928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.473162 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.473395 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.473396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.473396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.473504 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.473758 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:22.473764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:22.473889 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.473991 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.474136 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260928 02:43:22.474845 28847 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36119
I20260928 02:43:22.474934 28847 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:22.475104 28847 heartbeater.cc:511] Master 127.27.109.190:36119 requested a full tablet report, sending...
I20260928 02:43:22.475697 28656 ts_manager.cc:194] Registered new tserver with Master: 6ed39ea871f74e5db73aba4d0a9f9d1f (127.27.109.129:37811)
I20260928 02:43:22.476621 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:41943
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:22.492836 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:22.570860 28852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:22.571050 28852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:22.571074 28852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:22.572197 28852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:22.572239 28852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:22.572266 28852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:22.573673 28852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:22.573880 28852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:22.574066 28852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:22.578892 28857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:22.578846 28860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.579025 28852 server_base.cc:1034] running on GCE node
W20260928 02:43:22.578861 28858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.579345 28852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:22.579582 28852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:22.580740 28852 hybrid_clock.cc:648] HybridClock initialized: now 1790563402580737 us; error 36 us; skew 500 ppm
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:22.583158 28852 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:22.583606 28852 webserver.cc:460] Webserver started at http://127.27.109.130:36717/ using document root <none> and password file <none>
I20260928 02:43:22.583820 28852 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:22.583885 28852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:22.584028 28852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:22.585005 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "51408ec03d5149d3b38edee15ad2319a"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "9125482e863cd597bb6ea5e9f99851b6"
server_key_iv: "e6355707a4759905f776c346ed8f2394"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.585345 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "51408ec03d5149d3b38edee15ad2319a"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "9125482e863cd597bb6ea5e9f99851b6"
server_key_iv: "e6355707a4759905f776c346ed8f2394"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.586820 28852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:22.587584 28867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.587756 28852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:22.587852 28852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "51408ec03d5149d3b38edee15ad2319a"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "9125482e863cd597bb6ea5e9f99851b6"
server_key_iv: "e6355707a4759905f776c346ed8f2394"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.587937 28852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:22.594728 28852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:22.596511 28852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:22.596686 28852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:22.596905 28852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:22.597169 28852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:22.597223 28852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.597262 28852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:22.597290 28852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.602892 28852 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:36123
I20260928 02:43:22.602937 28980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:36123 every 8 connection(s)
I20260928 02:43:22.603201 28852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:22.606886 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28852
I20260928 02:43:22.606964 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:22.607120 28086 external_mini_cluster.cc:1428] Setting key bb0f6204ac16ffbd91448fc3d3b27b9c
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:22.612967 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.603757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36119 (local address 127.27.109.130:59263)
0928 02:43:22.603902 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.603933 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.604836 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.605103 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.605106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.605340 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:22.605508 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.605512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.606220 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.606221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.607137 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.607139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.607689 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.607690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.607800 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.608287 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:22.608295 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:22.609928 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:22.611727 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.611732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.611734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.611936 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.612147 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.612148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.612148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.612240 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.612534 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:22.612539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:22.612635 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.612721 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.612827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20260928 02:43:22.612986 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.603795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:59263 (local address 127.27.109.190:36119)
0928 02:43:22.603879 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.603881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.604017 (+ 136us) server_negotiation.cc:407] Connection header received
0928 02:43:22.604948 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.604949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.604962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.605027 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.605625 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.606126 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.607281 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.607546 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.607633 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.610024 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.610026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.610026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.610033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.611636 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.612014 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.612015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.612016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.612080 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.612307 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.612308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.612309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.612464 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.612543 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.612761 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.612848 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:43:22.613405 28981 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36119
I20260928 02:43:22.613495 28981 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:22.613675 28981 heartbeater.cc:511] Master 127.27.109.190:36119 requested a full tablet report, sending...
I20260928 02:43:22.614063 28656 ts_manager.cc:194] Registered new tserver with Master: 51408ec03d5149d3b38edee15ad2319a (127.27.109.130:36123)
I20260928 02:43:22.614598 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:59263
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:22.633369 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:22.711313 28986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:22.711542 28986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:22.711578 28986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:22.712694 28986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:22.712764 28986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:22.712807 28986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:22.714283 28986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:22.714556 28986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:22.714793 28986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:22.719446 28991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:22.719509 28994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:22.719444 28992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:22.719893 28986 server_base.cc:1034] running on GCE node
I20260928 02:43:22.720134 28986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:22.720366 28986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:22.721514 28986 hybrid_clock.cc:648] HybridClock initialized: now 1790563402721495 us; error 32 us; skew 500 ppm
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:22.724097 28986 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:22.724551 28986 webserver.cc:460] Webserver started at http://127.27.109.131:36891/ using document root <none> and password file <none>
I20260928 02:43:22.724786 28986 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:22.724829 28986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:22.724970 28986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:22.725946 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "30622278aaad4654ae291675bc8eb606"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "d130169daee8d3b947465406760dfe6f"
server_key_iv: "2e0000023df62b29b5fb440a82b934a8"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.726298 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "30622278aaad4654ae291675bc8eb606"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "d130169daee8d3b947465406760dfe6f"
server_key_iv: "2e0000023df62b29b5fb440a82b934a8"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.727638 28986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:22.728345 29001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.728554 28986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:22.728626 28986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "30622278aaad4654ae291675bc8eb606"
format_stamp: "Formatted at 2026-09-28 02:43:22 on dist-test-slave-b5p5"
server_key: "d130169daee8d3b947465406760dfe6f"
server_key_iv: "2e0000023df62b29b5fb440a82b934a8"
server_key_version: "encryptionkey@0"
I20260928 02:43:22.728705 28986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:22.750403 28986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:22.752717 28986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:22.752825 28986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:22.753005 28986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:22.753238 28986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:22.753268 28986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.753289 28986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:22.753301 28986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:22.758994 28986 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:46305
I20260928 02:43:22.759065 29114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:46305 every 8 connection(s)
I20260928 02:43:22.759332 28986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:22.767669 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 28986
I20260928 02:43:22.767792 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:22.767971 28086 external_mini_cluster.cc:1428] Setting key fb1a3cb784c2f9936d6c7e2c5c27d445
I20260928 02:43:22.768524 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.760004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:37787 (local address 127.27.109.190:36119)
0928 02:43:22.760069 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.760071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.760265 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:43:22.761101 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.761103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.761112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.761180 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.761764 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.762247 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.763341 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.763556 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.763611 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.765677 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.765679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.765679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.765686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.767099 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.767523 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.767524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.767524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.767581 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.767852 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.767853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.767854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.767995 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.768073 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.768336 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.768421 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:22.768592 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.759966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36119 (local address 127.27.109.131:37787)
0928 02:43:22.760115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.760160 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.761016 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.761247 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.761251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.761513 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:22.761688 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.761693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.762323 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.762324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.763268 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.763271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.763632 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.763633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.763725 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.764216 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:22.764225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:22.765582 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:22.767201 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.767208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.767210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.767426 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.767650 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:22.767651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:22.767651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:22.767761 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:22.768095 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:22.768100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:22.768217 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.768298 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.768407 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":100,"threads_started":1}
I20260928 02:43:22.769037 29115 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36119
I20260928 02:43:22.769135 29115 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:22.769321 29115 heartbeater.cc:511] Master 127.27.109.190:36119 requested a full tablet report, sending...
I20260928 02:43:22.769660 28656 ts_manager.cc:194] Registered new tserver with Master: 30622278aaad4654ae291675bc8eb606 (127.27.109.131:46305)
I20260928 02:43:22.770022 28656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:37787
I20260928 02:43:22.779539 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:22.786671 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.781030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.27.109.190:36119)
0928 02:43:22.781094 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.781097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.781159 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:43:22.781209 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.781210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.781218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.781261 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.781753 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.782184 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.782950 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.783118 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.783167 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.784165 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.784166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.784166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.784171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.785499 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.785891 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.785892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.785892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.785937 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.786140 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.786141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.786141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.786292 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.786362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.786458 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.786580 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:43:22.788733 28656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:22.790182 28656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:22.796185 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:36123 (local address 127.0.0.1:41048)
0928 02:43:22.792685 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.792694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.792748 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.792948 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.792949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.792951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:22.792996 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.792998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.794203 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.794204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.795114 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.795117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.795966 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.795967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.796023 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.796034 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.796102 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.796136 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":48,"threads_started":1}
I20260928 02:43:22.796253 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.27.109.130:36123)
0928 02:43:22.792628 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.792632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.792767 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:43:22.792843 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.792846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.792861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.792941 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:22.793054 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.794096 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.795198 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.795871 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.795948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.795998 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.796128 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.796165 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260928 02:43:22.796547 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37811 (local address 127.0.0.1:46440)
0928 02:43:22.792427 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.792438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.792500 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.793125 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.793126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.793127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:22.793150 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.793151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.794091 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.794093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.795424 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.795425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.796327 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.796328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.796384 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.796387 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.796436 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.796480 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":59,"thread_start_us":50,"threads_started":1}
I20260928 02:43:22.796809 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.27.109.129:37811)
0928 02:43:22.792860 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.792864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.792873 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:22.792930 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.792932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.792948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.793026 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:22.793285 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.793985 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.795508 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.796221 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.796291 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.796348 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.796695 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.796737 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":76,"threads_started":1}
I20260928 02:43:22.796988 28915 tablet_service.cc:1467] Processing CreateTablet for tablet 1aed3dbf8fe84eba8e214f37a72d0246 (DEFAULT_TABLE table=test-table [id=efb5d9896e97493cb7adbde0fc17e039]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:22.797336 28915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aed3dbf8fe84eba8e214f37a72d0246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:22.797426 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:46305 (local address 127.0.0.1:60444)
0928 02:43:22.792312 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.792438 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.792501 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.792710 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.792713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.792715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:22.792778 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.792780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.794172 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.794173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.795423 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.795425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.797165 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.797166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.797246 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.797249 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.797304 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.797347 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":38,"thread_start_us":61,"threads_started":1}
I20260928 02:43:22.797448 28781 tablet_service.cc:1467] Processing CreateTablet for tablet 1aed3dbf8fe84eba8e214f37a72d0246 (DEFAULT_TABLE table=test-table [id=efb5d9896e97493cb7adbde0fc17e039]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:22.797677 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.792224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.27.109.131:46305)
0928 02:43:22.792495 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.792500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.792511 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:22.792593 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.792595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.792608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.792675 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:22.792850 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.794083 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.796382 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.797084 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.797158 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.797204 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.797582 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.797625 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260928 02:43:22.797731 28781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aed3dbf8fe84eba8e214f37a72d0246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:22.798185 29049 tablet_service.cc:1467] Processing CreateTablet for tablet 1aed3dbf8fe84eba8e214f37a72d0246 (DEFAULT_TABLE table=test-table [id=efb5d9896e97493cb7adbde0fc17e039]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:22.798393 29049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aed3dbf8fe84eba8e214f37a72d0246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:22.800670 29129 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Bootstrap starting.
I20260928 02:43:22.800920 29130 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: Bootstrap starting.
I20260928 02:43:22.801148 29131 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: Bootstrap starting.
I20260928 02:43:22.801527 29129 tablet_bootstrap.cc:654] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:22.801587 29130 tablet_bootstrap.cc:654] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:22.801800 29131 tablet_bootstrap.cc:654] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:22.802054 29129 log.cc:824] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:22.802196 29130 log.cc:824] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:22.802369 29131 log.cc:824] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:22.802912 29130 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: No bootstrap required, opened a new log
I20260928 02:43:22.802912 29129 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: No bootstrap required, opened a new log
I20260928 02:43:22.802973 29129 ts_tablet_manager.cc:1397] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:22.802994 29130 ts_tablet_manager.cc:1397] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:22.803048 29131 tablet_bootstrap.cc:492] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: No bootstrap required, opened a new log
I20260928 02:43:22.803114 29131 ts_tablet_manager.cc:1397] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:22.804478 29130 raft_consensus.cc:348] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.804659 29130 raft_consensus.cc:374] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:22.804702 29130 raft_consensus.cc:729] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed39ea871f74e5db73aba4d0a9f9d1f, State: Initialized, Role: FOLLOWER
I20260928 02:43:22.804811 29130 consensus_queue.cc:260] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.804924 29131 raft_consensus.cc:348] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.804899 29129 raft_consensus.cc:348] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.805068 29131 raft_consensus.cc:374] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:22.805068 29129 raft_consensus.cc:374] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:22.805083 29130 ts_tablet_manager.cc:1428] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:22.805107 29129 raft_consensus.cc:729] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51408ec03d5149d3b38edee15ad2319a, State: Initialized, Role: FOLLOWER
I20260928 02:43:22.805107 29131 raft_consensus.cc:729] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30622278aaad4654ae291675bc8eb606, State: Initialized, Role: FOLLOWER
I20260928 02:43:22.805150 28847 heartbeater.cc:503] Master 127.27.109.190:36119 was elected leader, sending a full tablet report...
I20260928 02:43:22.805207 29129 consensus_queue.cc:260] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.805217 29131 consensus_queue.cc:260] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.805455 29129 ts_tablet_manager.cc:1428] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:22.805454 29131 ts_tablet_manager.cc:1428] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:22.805600 28981 heartbeater.cc:503] Master 127.27.109.190:36119 was elected leader, sending a full tablet report...
I20260928 02:43:22.805591 29115 heartbeater.cc:503] Master 127.27.109.190:36119 was elected leader, sending a full tablet report...
I20260928 02:43:22.808027 29135 raft_consensus.cc:482] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:22.808089 29135 raft_consensus.cc:504] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.808382 29135 leader_election.cc:288] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30622278aaad4654ae291675bc8eb606 (127.27.109.131:46305), 51408ec03d5149d3b38edee15ad2319a (127.27.109.130:36123)
I20260928 02:43:22.811187 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.808591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:39673 (local address 127.27.109.131:46305)
0928 02:43:22.808746 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.808748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.808755 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:22.808773 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.808774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.808782 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.808858 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:22.809016 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.809480 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810458 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.810834 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810928 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.810935 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.811093 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.811125 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 02:43:22.811211 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.808663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:35769 (local address 127.27.109.130:36123)
0928 02:43:22.808709 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.808711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.808829 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:43:22.808861 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.808862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.808869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.808917 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:22.809082 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.809574 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810513 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.810866 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810962 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.810969 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.811121 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.811152 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:43:22.811220 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.808513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:46305 (local address 127.27.109.129:39673)
0928 02:43:22.808548 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.808559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.808595 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.808899 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.808900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.808902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:22.808957 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.808959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.809561 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.809562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.810350 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.810352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.810928 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.811003 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.811008 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.811087 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.811125 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 02:43:22.811280 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.808609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:36123 (local address 127.27.109.129:35769)
0928 02:43:22.808759 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:22.808768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:22.808820 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:22.808938 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:22.808938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:22.808940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:22.808995 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.808996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.809652 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.809653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.810380 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:22.810382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.810963 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.810964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:22.811036 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.811039 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:22.811113 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:43:22.811151 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260928 02:43:22.811509 29069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aed3dbf8fe84eba8e214f37a72d0246" candidate_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30622278aaad4654ae291675bc8eb606" is_pre_election: true
I20260928 02:43:22.811529 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aed3dbf8fe84eba8e214f37a72d0246" candidate_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51408ec03d5149d3b38edee15ad2319a" is_pre_election: true
I20260928 02:43:22.811659 29069 raft_consensus.cc:2454] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed39ea871f74e5db73aba4d0a9f9d1f in term 0.
I20260928 02:43:22.811677 28935 raft_consensus.cc:2454] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed39ea871f74e5db73aba4d0a9f9d1f in term 0.
I20260928 02:43:22.811858 28734 leader_election.cc:302] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30622278aaad4654ae291675bc8eb606, 6ed39ea871f74e5db73aba4d0a9f9d1f; no voters:
I20260928 02:43:22.812041 29135 raft_consensus.cc:2781] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:22.812098 29135 raft_consensus.cc:482] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:22.812132 29135 raft_consensus.cc:3037] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:22.812695 29135 raft_consensus.cc:504] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.812844 29135 leader_election.cc:288] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [CANDIDATE]: Term 1 election: Requested vote from peers 30622278aaad4654ae291675bc8eb606 (127.27.109.131:46305), 51408ec03d5149d3b38edee15ad2319a (127.27.109.130:36123)
I20260928 02:43:22.812996 29069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aed3dbf8fe84eba8e214f37a72d0246" candidate_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30622278aaad4654ae291675bc8eb606"
I20260928 02:43:22.812995 28935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aed3dbf8fe84eba8e214f37a72d0246" candidate_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51408ec03d5149d3b38edee15ad2319a"
I20260928 02:43:22.813071 29069 raft_consensus.cc:3037] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:22.813076 28935 raft_consensus.cc:3037] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:22.813675 29069 raft_consensus.cc:2454] T 1aed3dbf8fe84eba8e214f37a72d0246 P 30622278aaad4654ae291675bc8eb606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed39ea871f74e5db73aba4d0a9f9d1f in term 1.
I20260928 02:43:22.813688 28935 raft_consensus.cc:2454] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed39ea871f74e5db73aba4d0a9f9d1f in term 1.
I20260928 02:43:22.813834 28734 leader_election.cc:302] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51408ec03d5149d3b38edee15ad2319a, 6ed39ea871f74e5db73aba4d0a9f9d1f; no voters:
I20260928 02:43:22.813918 29135 raft_consensus.cc:2781] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:22.814064 29135 raft_consensus.cc:686] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [term 1 LEADER]: Becoming Leader. State: Replica: 6ed39ea871f74e5db73aba4d0a9f9d1f, State: Running, Role: LEADER
I20260928 02:43:22.814178 29135 consensus_queue.cc:237] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } }
I20260928 02:43:22.814934 28656 catalog_manager.cc:5495] T 1aed3dbf8fe84eba8e214f37a72d0246 P 6ed39ea871f74e5db73aba4d0a9f9d1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed39ea871f74e5db73aba4d0a9f9d1f (127.27.109.129). New cstate: current_term: 1 leader_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30622278aaad4654ae291675bc8eb606" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 46305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51408ec03d5149d3b38edee15ad2319a" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ed39ea871f74e5db73aba4d0a9f9d1f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37811 } health_report { overall_health: HEALTHY } } }
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:22.827946 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.820883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.27.109.129:37811)
0928 02:43:22.820950 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.820952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.821069 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:43:22.821143 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.821143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.821153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.821219 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.821728 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.822154 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.822910 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.823095 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.823174 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.825432 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.825440 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.825440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.825447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.826860 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.827191 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.827194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.827194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.827251 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.827444 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.827445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.827446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.827568 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.827749 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.827784 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.827853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:22.849498 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.843085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.27.109.129:37811)
0928 02:43:22.843168 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.843169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.843263 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:43:22.843315 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.843315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.843323 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.843369 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.843927 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.844367 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.845059 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.845241 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.845292 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.847324 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.847325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.847325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.847330 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.848619 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.848909 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.848910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.848910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.848954 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.849113 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.849114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.849114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.849189 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.849259 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.849363 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.849413 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:43:22.853698 28982 tablet.cc:2369] T 1aed3dbf8fe84eba8e214f37a72d0246 P 51408ec03d5149d3b38edee15ad2319a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:22.870802 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:22.863898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.27.109.129:37811)
0928 02:43:22.864058 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:22.864061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:22.864283 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:43:22.864306 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:22.864307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:22.864316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:22.864381 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:22.864908 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.865356 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.866038 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:22.866228 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:22.866309 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:22.868798 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:22.868799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:22.868799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:22.868804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:22.869893 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.870176 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.870176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.870177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.870220 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:22.870363 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:22.870363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:22.870363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:22.870439 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:22.870616 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:22.870664 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:43:22.870718 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260928 02:43:22.871248 28761 tablet_service.cc:2908] Rejecting scan request for tablet 1aed3dbf8fe84eba8e214f37a72d0246: Uninitialized: clean time has not yet been initialized
I20260928 02:43:22.871958 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28718
I20260928 02:43:22.877000 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28852
I20260928 02:43:22.881181 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28986
I20260928 02:43:22.885210 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 28624
2026-09-28T02:43:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1859 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[29153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[29153](info): set up 2 sockets
Sep 28 02:43:22 dist-test-slave-b5p5 krb5kdc[29153](info): commencing operation
krb5kdc: starting...
W20260928 02:43:23.910413 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:23 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:24.020961 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:41213
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:24.099215 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:24.099429 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:24.099466 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:24.100581 29166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:24.100641 29166 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:43:24.100667 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:24.100701 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:24.100739 29166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:24.100767 29166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:24.102169 29166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:41213
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:24.102416 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:24.102638 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:43:24.107329 29166 server_base.cc:1034] running on GCE node
W20260928 02:43:24.107303 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.107291 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.107470 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:24.107749 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:24.108021 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:24.109180 29166 hybrid_clock.cc:648] HybridClock initialized: now 1790563404109166 us; error 44 us; skew 500 ppm
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:24.111604 29166 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:24.112025 29166 webserver.cc:460] Webserver started at http://127.27.109.190:46205/ using document root <none> and password file <none>
I20260928 02:43:24.112246 29166 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:24.112313 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:24.112449 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:24.113426 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "8f81d731cf7346e6aa9c0ccd7e806d79"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "890750c32c7fcce5b74f546fe8478fb8"
server_key_iv: "3453f8e2ea0ac5e5452b08594fa3299d"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.113775 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "8f81d731cf7346e6aa9c0ccd7e806d79"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "890750c32c7fcce5b74f546fe8478fb8"
server_key_iv: "3453f8e2ea0ac5e5452b08594fa3299d"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.115115 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:24.115799 29181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.115976 29166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:24.116070 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "8f81d731cf7346e6aa9c0ccd7e806d79"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "890750c32c7fcce5b74f546fe8478fb8"
server_key_iv: "3453f8e2ea0ac5e5452b08594fa3299d"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.116142 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:24.132634 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:24.134832 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:24.134984 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:24.139365 29166 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:41213
I20260928 02:43:24.139437 29243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:41213 every 8 connection(s)
I20260928 02:43:24.139739 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:24.140375 29244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.142530 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Bootstrap starting.
I20260928 02:43:24.143088 29244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.143369 29244 log.cc:824] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:24.144012 29244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: No bootstrap required, opened a new log
I20260928 02:43:24.144816 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29166
I20260928 02:43:24.144927 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:24.145078 28086 external_mini_cluster.cc:1428] Setting key a32d7ae90655e6cf9d657e45c26da592
I20260928 02:43:24.146647 29244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } }
I20260928 02:43:24.146817 29244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.146853 29244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f81d731cf7346e6aa9c0ccd7e806d79, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.146967 29244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } }
I20260928 02:43:24.147046 29244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:24.147069 29244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:24.147137 29244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.147712 29244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } }
I20260928 02:43:24.147815 29244 leader_election.cc:302] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f81d731cf7346e6aa9c0ccd7e806d79; no voters:
I20260928 02:43:24.147951 29244 leader_election.cc:288] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:24.148065 29249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:24.148252 29244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:24.148263 29249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [term 1 LEADER]: Becoming Leader. State: Replica: 8f81d731cf7346e6aa9c0ccd7e806d79, State: Running, Role: LEADER
I20260928 02:43:24.148389 29249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } }
I20260928 02:43:24.148736 29250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } } }
I20260928 02:43:24.148833 29250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:24.148736 29251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f81d731cf7346e6aa9c0ccd7e806d79. Latest consensus state: current_term: 1 leader_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f81d731cf7346e6aa9c0ccd7e806d79" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 41213 } } }
I20260928 02:43:24.149230 29251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:24.149461 29258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:24.149962 29258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.156822 29258 catalog_manager.cc:1347] Generated new cluster ID: b5c430e1e65344bba2f4b9d1d3d8defc
I20260928 02:43:24.156888 29258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:24.157708 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.145564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.27.109.190:41213)
0928 02:43:24.145794 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.145806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.145844 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:43:24.146610 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.146618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.146680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.146841 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.147609 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.149237 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.150356 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.150894 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.151019 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.153585 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.153590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.153590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.153602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.156055 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.156458 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.156461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.156462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.156581 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.156797 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.156798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.156798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.157094 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.157245 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.157366 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.157505 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260928 02:43:24.168380 29258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:24.169064 29258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:24.179867 29258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Generated new TSK 0
I20260928 02:43:24.180052 29258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:24.210868 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:41213
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:24.304579 29270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:24.304770 29270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:24.304793 29270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:24.305925 29270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:24.305964 29270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:24.305984 29270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:24.306001 29270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:24.307423 29270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:24.307654 29270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:24.307898 29270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:24.312712 29275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.312808 29276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:24.312949 29270 server_base.cc:1034] running on GCE node
W20260928 02:43:24.312992 29278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:24.313210 29270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:24.313409 29270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:24.314529 29270 hybrid_clock.cc:648] HybridClock initialized: now 1790563404314516 us; error 27 us; skew 500 ppm
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:24.320050 29270 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:24.320483 29270 webserver.cc:460] Webserver started at http://127.27.109.129:35477/ using document root <none> and password file <none>
I20260928 02:43:24.320744 29270 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:24.320813 29270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:24.320948 29270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:24.321981 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "d9360eb71de4c3c13154a919f28760dd"
server_key_iv: "30951444e6c7898bf84c2d5317eecfa3"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.322333 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "d9360eb71de4c3c13154a919f28760dd"
server_key_iv: "30951444e6c7898bf84c2d5317eecfa3"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.323625 29270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.324337 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.324503 29270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.324611 29270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "d9360eb71de4c3c13154a919f28760dd"
server_key_iv: "30951444e6c7898bf84c2d5317eecfa3"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.324697 29270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:24.346210 29270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:24.348628 29270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:24.348800 29270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:24.349282 29270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:24.349380 29270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.349411 29270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:24.349433 29270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.356416 29270 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:40237
I20260928 02:43:24.356505 29400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:40237 every 8 connection(s)
I20260928 02:43:24.356799 29270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:24.359275 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.349742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:37015 (local address 127.27.109.190:41213)
0928 02:43:24.349783 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.349785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.350000 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:43:24.350523 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.350525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.350541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.350606 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.351347 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.352153 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.353809 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.354152 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.354234 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.356875 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.356876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.356877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.356886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.358125 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.358567 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.358568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.358568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.358628 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.358817 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.358818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.358818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.358912 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.358959 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.359157 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.359216 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:24.359427 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.349726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.129:37015)
0928 02:43:24.349888 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.349920 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.350398 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.350675 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.350679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.350980 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.351264 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.351270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.352257 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.352258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.353705 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.353708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.354257 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.354258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.354424 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.354919 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.354930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.356781 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.358194 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.358199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.358200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.358438 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.358671 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.358672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.358672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.358759 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.358965 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.358969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.359067 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.359137 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.359248 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 02:43:24.359906 29401 heartbeater.cc:347] Connected to a master server at 127.27.109.190:41213
I20260928 02:43:24.360003 29401 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:24.360170 29401 heartbeater.cc:511] Master 127.27.109.190:41213 requested a full tablet report, sending...
I20260928 02:43:24.360667 29197 ts_manager.cc:194] Registered new tserver with Master: b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237)
I20260928 02:43:24.361538 29197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:37015
I20260928 02:43:24.365834 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.360755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.129:39393)
0928 02:43:24.360872 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.360883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.360936 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.361045 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.361046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.361231 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.361283 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.361284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.361970 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.361971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.362584 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.362585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.362963 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.362964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.363036 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.363362 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.363366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.363604 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.364840 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.364841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.364841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.365006 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.365219 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.365219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.365220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.365288 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.365519 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.365520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.365562 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.365650 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.365743 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":16,"thread_start_us":61,"threads_started":1}
I20260928 02:43:24.365855 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.360818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:39393 (local address 127.27.109.190:41213)
0928 02:43:24.360855 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.360857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.360923 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:43:24.360988 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.360989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.360996 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.361040 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.361342 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.361876 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.362672 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.362880 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.362962 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.363704 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.363705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.363705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.363711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.364761 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.365090 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.365091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.365091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.365146 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.365356 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.365357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.365357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.365457 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.365527 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.365672 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.365773 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:24.365901 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29270
I20260928 02:43:24.366008 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:24.366178 28086 external_mini_cluster.cc:1428] Setting key f31c249d37cee9eb1b7e8333d8ad4af7
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:24.389158 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:41213
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:24.468045 29410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:24.468237 29410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:24.468261 29410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:24.469411 29410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:24.469451 29410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:24.469498 29410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:24.469534 29410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:24.470935 29410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:24.471169 29410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:24.471386 29410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:24.476145 29416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.476158 29415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.476159 29418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:24.476344 29410 server_base.cc:1034] running on GCE node
I20260928 02:43:24.476567 29410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:24.476794 29410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:24.477957 29410 hybrid_clock.cc:648] HybridClock initialized: now 1790563404477937 us; error 30 us; skew 500 ppm
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:24.480239 29410 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:24.480677 29410 webserver.cc:460] Webserver started at http://127.27.109.130:35495/ using document root <none> and password file <none>
I20260928 02:43:24.480902 29410 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:24.480968 29410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:24.481114 29410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:24.482086 29410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "66f0b0bd292a4fae85088c28f52fa89d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "03579812f5e6a5fccbbb52933ab38eb5"
server_key_iv: "c54f52061eed4cdbd12fe4aeb5968d85"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.482434 29410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "66f0b0bd292a4fae85088c28f52fa89d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "03579812f5e6a5fccbbb52933ab38eb5"
server_key_iv: "c54f52061eed4cdbd12fe4aeb5968d85"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.483762 29410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.484547 29425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.484735 29410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.484836 29410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "66f0b0bd292a4fae85088c28f52fa89d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "03579812f5e6a5fccbbb52933ab38eb5"
server_key_iv: "c54f52061eed4cdbd12fe4aeb5968d85"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.484920 29410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:24.511921 29410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:24.514012 29410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:24.514180 29410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:24.514878 29410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:24.514930 29410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.514966 29410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:24.515015 29410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.522013 29410 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:38923
I20260928 02:43:24.522131 29540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:38923 every 8 connection(s)
I20260928 02:43:24.522426 29410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:24.523615 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29410
I20260928 02:43:24.523716 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:24.523890 28086 external_mini_cluster.cc:1428] Setting key 297db238dfcc8fd6e19178b91099a49f
I20260928 02:43:24.525542 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.515264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:46235 (local address 127.27.109.190:41213)
0928 02:43:24.515323 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.515326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.515554 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:43:24.516113 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.516114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.516127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.516183 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.516861 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.517392 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.518668 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.519016 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.519107 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.522345 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.522346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.522346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.522354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.524080 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.524580 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.524581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.524582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.524654 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.524906 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.524907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.524907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.525025 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.525101 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.525366 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.525452 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:43:24.525718 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.515272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.130:46235)
0928 02:43:24.515441 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.515481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.515998 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.516259 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.516263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.516557 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.516755 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.516759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.517484 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.517486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.518570 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.518572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.519111 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.519112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.519255 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.519771 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.519788 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.522251 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.524228 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.524236 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.524238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.524502 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.524730 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.524731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.524731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.524839 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.525090 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.525096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.525237 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.525345 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.525498 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260928 02:43:24.526280 29541 heartbeater.cc:347] Connected to a master server at 127.27.109.190:41213
I20260928 02:43:24.526374 29541 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:24.526588 29541 heartbeater.cc:511] Master 127.27.109.190:41213 requested a full tablet report, sending...
I20260928 02:43:24.526965 29197 ts_manager.cc:194] Registered new tserver with Master: 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130:38923)
I20260928 02:43:24.527469 29197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:46235
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:43:24.533761 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.527824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:46775 (local address 127.27.109.190:41213)
0928 02:43:24.527860 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.527862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.527991 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:43:24.528035 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.528036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.528045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.528104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.528473 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.528973 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.530120 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.530289 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.530338 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.531328 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.531330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.531330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.531336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.532830 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.533121 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.533122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.533122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.533165 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.533305 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.533306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.533306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.533452 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.533518 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.533637 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.533700 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:24.533766 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.527780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.130:46775)
0928 02:43:24.527920 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.527931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.527986 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.528106 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.528107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.528330 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.528404 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.528406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.529076 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.529078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.530031 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.530034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.530362 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.530363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.530441 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.530823 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.530828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.531222 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.532908 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.532909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.532910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.533062 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.533220 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.533221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.533221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.533268 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.533515 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.533516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.533551 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.533601 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.533703 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:24.553094 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:41213
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:24.632544 29550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:24.632758 29550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:24.632783 29550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:24.633944 29550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:24.633985 29550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:24.634032 29550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:24.634068 29550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:24.635478 29550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:41213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:24.635703 29550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:24.635922 29550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:24.640658 29558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.640720 29555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:24.640805 29556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:24.640918 29550 server_base.cc:1034] running on GCE node
I20260928 02:43:24.641206 29550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:24.641438 29550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:24.642612 29550 hybrid_clock.cc:648] HybridClock initialized: now 1790563404642594 us; error 24 us; skew 500 ppm
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:24.645254 29550 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:24.645720 29550 webserver.cc:460] Webserver started at http://127.27.109.131:36223/ using document root <none> and password file <none>
I20260928 02:43:24.645943 29550 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:24.646009 29550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:24.646173 29550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:24.647279 29550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "bdaf08a0c08e1835bdd4362ea9eb4130"
server_key_iv: "6dd322b574a60b862634e81873a3b119"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.647640 29550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "bdaf08a0c08e1835bdd4362ea9eb4130"
server_key_iv: "6dd322b574a60b862634e81873a3b119"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.648970 29550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:43:24.649782 29565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.649953 29550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:24.650050 29550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d"
format_stamp: "Formatted at 2026-09-28 02:43:24 on dist-test-slave-b5p5"
server_key: "bdaf08a0c08e1835bdd4362ea9eb4130"
server_key_iv: "6dd322b574a60b862634e81873a3b119"
server_key_version: "encryptionkey@0"
I20260928 02:43:24.650141 29550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:24.674557 29550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:24.676898 29550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:24.677063 29550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:24.677666 29550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:24.677717 29550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.677754 29550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:24.677807 29550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.685025 29550 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39959
I20260928 02:43:24.685097 29680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39959 every 8 connection(s)
I20260928 02:43:24.685461 29550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:24.687119 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:54337 (local address 127.27.109.190:41213)
0928 02:43:24.678124 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.678127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.678348 (+ 221us) server_negotiation.cc:407] Connection header received
0928 02:43:24.678900 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.678901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.678915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.679005 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.679565 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.680271 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.681418 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.681620 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.681694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.684399 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.684401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.684401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.684410 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.686042 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.686380 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.686381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.686381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.686429 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.686669 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.686669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.686670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.686777 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.686827 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.687003 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.687058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:24.687322 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.131:54337)
0928 02:43:24.678217 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.678258 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.678802 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.679058 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.679062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.679284 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.679465 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.679469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.680387 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.680388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.681322 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.681325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.681698 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.681699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.681827 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.682298 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.682308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.684275 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.686109 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.686113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.686114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.686314 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.686518 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.686518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.686519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.686616 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.686819 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.686821 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.686916 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.686982 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.687120 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260928 02:43:24.687697 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29550
I20260928 02:43:24.687777 29681 heartbeater.cc:347] Connected to a master server at 127.27.109.190:41213
I20260928 02:43:24.687795 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:24.687852 29681 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:24.687963 28086 external_mini_cluster.cc:1428] Setting key 9785228aeaa4321f97fe1c0483c16b1a
I20260928 02:43:24.688020 29681 heartbeater.cc:511] Master 127.27.109.190:41213 requested a full tablet report, sending...
I20260928 02:43:24.688375 29198 ts_manager.cc:194] Registered new tserver with Master: 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959)
I20260928 02:43:24.688803 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:24.688886 29198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:54337
I20260928 02:43:24.696292 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.688941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:50499 (local address 127.27.109.190:41213)
0928 02:43:24.690454 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.690456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.690509 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:43:24.690527 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.690528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.690536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.690607 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.690911 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.691357 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.692423 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.692599 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.692696 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.693597 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.693598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.693598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.693604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.695219 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.695600 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.695601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.695601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.695647 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.695888 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.695888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.695888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.695976 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.696023 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.696189 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.696239 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494}
I20260928 02:43:24.696437 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.688946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.27.109.131:50499)
0928 02:43:24.689208 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.689220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.689275 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.690612 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.690613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.690782 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.690827 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.690829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.691446 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.691447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.692339 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.692341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.692684 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.692685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.692768 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.693134 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.693140 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.693507 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.695294 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.695296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.695296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.695524 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.695739 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.695740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.695741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.695821 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.696038 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.696039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.696089 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.696165 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.696284 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":73,"threads_started":1}
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.711114 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.705165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.27.109.190:41213)
0928 02:43:24.705220 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.705222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.705232 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:24.705249 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.705249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.705256 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.705301 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.705805 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.706225 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.706943 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.707108 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.707156 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.709081 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.709082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.709083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.709088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.710185 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.710518 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.710519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.710519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.710564 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.710739 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.710739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.710739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.710824 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.710891 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.710988 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.711038 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:43:24.712658 29198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:24.714140 29198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:24.720321 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38923 (local address 127.0.0.1:59266)
0928 02:43:24.716460 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.716497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.716555 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.716787 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.716790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.716793 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.716860 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.716862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.717875 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.717876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.719094 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.719096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.720011 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.720088 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720101 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.720179 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.720248 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260928 02:43:24.720350 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.27.109.130:38923)
0928 02:43:24.716606 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.716611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.716622 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:24.716661 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.716664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.716679 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.716780 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.716926 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.717789 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.719188 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.719929 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720004 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720057 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.720201 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.720239 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260928 02:43:24.720665 29698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39959 (local address 127.0.0.1:45348)
0928 02:43:24.716591 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.716600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.716638 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.717011 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.717012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.717014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.717115 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.717117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.718374 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.718376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.719479 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.719482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.720423 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.720503 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720506 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.720563 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.720598 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 02:43:24.720714 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.27.109.131:39959)
0928 02:43:24.716870 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.716874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.716882 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:24.716914 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.716915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.716929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.717002 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.717209 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.718288 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.719562 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.720270 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720338 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720387 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.720579 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.720619 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":82,"threads_started":1}
I20260928 02:43:24.721055 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40237 (local address 127.0.0.1:59338)
0928 02:43:24.716708 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.716718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.716771 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.717106 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.717106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.717108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.717138 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.717140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.718410 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.718411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.719442 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.719444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.720333 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.720665 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720671 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.720949 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.720987 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":65,"threads_started":1}
I20260928 02:43:24.721038 29475 tablet_service.cc:1467] Processing CreateTablet for tablet 247cf54f5ca04006b41b10459ed6b561 (DEFAULT_TABLE table=test-table [id=702634611e2e4f9ebba074271df5640d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.721390 29475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247cf54f5ca04006b41b10459ed6b561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.721417 29615 tablet_service.cc:1467] Processing CreateTablet for tablet 247cf54f5ca04006b41b10459ed6b561 (DEFAULT_TABLE table=test-table [id=702634611e2e4f9ebba074271df5640d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.721720 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.716718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.27.109.129:40237)
0928 02:43:24.716903 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.716908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.716917 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:24.716957 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.716959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.716973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.717045 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.717213 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.718311 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.719523 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.720189 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.720250 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.720300 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.721543 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.721614 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260928 02:43:24.721755 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247cf54f5ca04006b41b10459ed6b561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.724081 29704 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Bootstrap starting.
I20260928 02:43:24.724400 29705 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Bootstrap starting.
I20260928 02:43:24.724740 29335 tablet_service.cc:1467] Processing CreateTablet for tablet 247cf54f5ca04006b41b10459ed6b561 (DEFAULT_TABLE table=test-table [id=702634611e2e4f9ebba074271df5640d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.724961 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 247cf54f5ca04006b41b10459ed6b561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.724967 29704 tablet_bootstrap.cc:654] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.725005 29705 tablet_bootstrap.cc:654] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.725299 29705 log.cc:824] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:24.725306 29704 log.cc:824] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:24.726049 29705 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: No bootstrap required, opened a new log
I20260928 02:43:24.726092 29705 ts_tablet_manager.cc:1397] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:24.726275 29704 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: No bootstrap required, opened a new log
I20260928 02:43:24.726336 29704 ts_tablet_manager.cc:1397] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:24.727867 29705 raft_consensus.cc:348] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.728017 29705 raft_consensus.cc:374] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.728053 29705 raft_consensus.cc:729] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f0b0bd292a4fae85088c28f52fa89d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.728156 29705 consensus_queue.cc:260] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.728406 29705 ts_tablet_manager.cc:1428] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:43:24.728360 29708 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Bootstrap starting.
I20260928 02:43:24.728513 29541 heartbeater.cc:503] Master 127.27.109.190:41213 was elected leader, sending a full tablet report...
I20260928 02:43:24.728621 29704 raft_consensus.cc:348] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.728786 29704 raft_consensus.cc:374] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.728829 29704 raft_consensus.cc:729] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607f4c5f7a0a4cb4a78b333fc1a04a8d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.728940 29704 consensus_queue.cc:260] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.728992 29708 tablet_bootstrap.cc:654] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.729189 29704 ts_tablet_manager.cc:1428] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:24.729199 29681 heartbeater.cc:503] Master 127.27.109.190:41213 was elected leader, sending a full tablet report...
I20260928 02:43:24.729368 29708 log.cc:824] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:24.730031 29708 tablet_bootstrap.cc:492] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: No bootstrap required, opened a new log
I20260928 02:43:24.730096 29708 ts_tablet_manager.cc:1397] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:24.731452 29708 raft_consensus.cc:348] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.731586 29708 raft_consensus.cc:374] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.731629 29708 raft_consensus.cc:729] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64a1b4de4334ec2ad7db0c5b6085054, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.731786 29708 consensus_queue.cc:260] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.732051 29708 ts_tablet_manager.cc:1428] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:43:24.732059 29401 heartbeater.cc:503] Master 127.27.109.190:41213 was elected leader, sending a full tablet report...
I20260928 02:43:24.739012 29709 raft_consensus.cc:482] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:24.739105 29709 raft_consensus.cc:504] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.739370 29709 leader_election.cc:288] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959), b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237)
I20260928 02:43:24.742952 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.739721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:48613 (local address 127.27.109.129:40237)
0928 02:43:24.739756 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.739757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.739823 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:43:24.739870 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.739871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.739878 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.739923 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.740038 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.740702 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742048 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.742613 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742687 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.742694 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.742858 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.742891 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:24.742952 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.739578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:55393 (local address 127.27.109.131:39959)
0928 02:43:24.739634 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.739635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.739645 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:24.739765 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.739766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.739775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.739830 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.739966 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.740665 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742048 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.742610 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742687 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.742694 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.742865 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.742895 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:43:24.742969 29434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.739530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39959 (local address 127.27.109.130:55393)
0928 02:43:24.739581 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.739590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.739651 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.739843 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.739844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.739846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.739903 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.739904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.740761 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.740762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.741960 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.741962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.742692 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.742799 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.742801 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.742854 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.742880 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":31}
I20260928 02:43:24.742978 29713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.739645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40237 (local address 127.27.109.130:48613)
0928 02:43:24.739755 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.739764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.739822 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.739926 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.739926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.739927 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.739983 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.739984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.740767 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.740767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.741960 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.741962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.742692 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.742693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.742767 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.742771 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.742842 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.742880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260928 02:43:24.743287 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247cf54f5ca04006b41b10459ed6b561" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" is_pre_election: true
I20260928 02:43:24.743415 29635 raft_consensus.cc:2454] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 0.
I20260928 02:43:24.743626 29429 leader_election.cc:302] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, 66f0b0bd292a4fae85088c28f52fa89d; no voters:
I20260928 02:43:24.743633 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247cf54f5ca04006b41b10459ed6b561" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" is_pre_election: true
I20260928 02:43:24.743736 29355 raft_consensus.cc:2454] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 0.
I20260928 02:43:24.743790 29709 raft_consensus.cc:2781] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:24.743860 29709 raft_consensus.cc:482] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:24.743896 29709 raft_consensus.cc:3037] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.744712 29709 raft_consensus.cc:504] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.744845 29709 leader_election.cc:288] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 election: Requested vote from peers 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959), b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237)
I20260928 02:43:24.745028 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247cf54f5ca04006b41b10459ed6b561" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d"
I20260928 02:43:24.745064 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247cf54f5ca04006b41b10459ed6b561" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
I20260928 02:43:24.745111 29635 raft_consensus.cc:3037] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.745162 29355 raft_consensus.cc:3037] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.745734 29635 raft_consensus.cc:2454] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 1.
I20260928 02:43:24.745793 29355 raft_consensus.cc:2454] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 1.
I20260928 02:43:24.745940 29429 leader_election.cc:302] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, 66f0b0bd292a4fae85088c28f52fa89d; no voters:
I20260928 02:43:24.746049 29709 raft_consensus.cc:2781] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:24.746217 29709 raft_consensus.cc:686] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 LEADER]: Becoming Leader. State: Replica: 66f0b0bd292a4fae85088c28f52fa89d, State: Running, Role: LEADER
I20260928 02:43:24.746322 29709 consensus_queue.cc:237] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } }
I20260928 02:43:24.747098 29197 catalog_manager.cc:5495] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130). New cstate: current_term: 1 leader_uuid: "66f0b0bd292a4fae85088c28f52fa89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } health_report { overall_health: HEALTHY } } }
I20260928 02:43:24.756366 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.753783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.27.109.130:38923)
0928 02:43:24.753840 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.753842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.753881 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:43:24.753976 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.753977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.753993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.754074 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:24.754222 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.754752 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.755544 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.755756 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.755838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.756038 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:24.756126 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:24.756215 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.756257 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.756275 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:43:24.757820 29635 raft_consensus.cc:1261] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Refusing update from remote peer 66f0b0bd292a4fae85088c28f52fa89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:24.758064 29355 raft_consensus.cc:1261] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Refusing update from remote peer 66f0b0bd292a4fae85088c28f52fa89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:24.758083 29709 consensus_queue.cc:1035] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:24.758370 29709 consensus_queue.cc:1035] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:24.760524 29717 mvcc.cc:205] Tried to move back new op lower bound from 7334147705885810688 to 7334147705841680384. Current Snapshot: MvccSnapshot[applied={T|T < 7334147705885810688}]
I20260928 02:43:24.761179 29720 mvcc.cc:205] Tried to move back new op lower bound from 7334147705885810688 to 7334147705841680384. Current Snapshot: MvccSnapshot[applied={T|T < 7334147705885810688}]
I20260928 02:43:24.763832 29198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:24.763926 29198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:24.764593 29198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Sending DeleteTablet for 3 replicas of tablet 247cf54f5ca04006b41b10459ed6b561
I20260928 02:43:24.764914 29475 tablet_service.cc:1514] Processing DeleteTablet for tablet 247cf54f5ca04006b41b10459ed6b561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:59266
I20260928 02:43:24.764972 29335 tablet_service.cc:1514] Processing DeleteTablet for tablet 247cf54f5ca04006b41b10459ed6b561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 02:43:24.764976 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet 247cf54f5ca04006b41b10459ed6b561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45348
I20260928 02:43:24.765321 29728 tablet_replica.cc:330] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: stopping tablet replica
I20260928 02:43:24.765453 29728 raft_consensus.cc:2229] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:24.765450 29198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:24.765571 29198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:24.766002 29728 raft_consensus.cc:2258] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:24.766225 29729 tablet_replica.cc:330] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: stopping tablet replica
I20260928 02:43:24.766322 29729 raft_consensus.cc:2229] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:24.766417 29729 raft_consensus.cc:2258] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:24.766829 29728 ts_tablet_manager.cc:1905] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:24.766920 29615 tablet_service.cc:1467] Processing CreateTablet for tablet faf09d41f9064322a9a3432d0f6bce45 (DEFAULT_TABLE table=test-table [id=78bd06a099a5444f8754fd04eb3266a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.766921 29335 tablet_service.cc:1467] Processing CreateTablet for tablet faf09d41f9064322a9a3432d0f6bce45 (DEFAULT_TABLE table=test-table [id=78bd06a099a5444f8754fd04eb3266a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.767022 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf09d41f9064322a9a3432d0f6bce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.767022 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf09d41f9064322a9a3432d0f6bce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.768046 29475 tablet_service.cc:1467] Processing CreateTablet for tablet faf09d41f9064322a9a3432d0f6bce45 (DEFAULT_TABLE table=test-table [id=78bd06a099a5444f8754fd04eb3266a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:24.768141 29475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf09d41f9064322a9a3432d0f6bce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.768391 29708 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Bootstrap starting.
I20260928 02:43:24.768756 29708 tablet_bootstrap.cc:654] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.769161 29728 ts_tablet_manager.cc:1918] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:24.769315 29708 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: No bootstrap required, opened a new log
I20260928 02:43:24.769388 29728 log.cc:1196] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/247cf54f5ca04006b41b10459ed6b561
I20260928 02:43:24.769414 29708 ts_tablet_manager.cc:1397] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.769567 29705 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Bootstrap starting.
I20260928 02:43:24.769554 29708 raft_consensus.cc:348] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.769635 29708 raft_consensus.cc:374] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.769690 29708 raft_consensus.cc:729] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64a1b4de4334ec2ad7db0c5b6085054, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.769738 29708 consensus_queue.cc:260] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.769838 29708 ts_tablet_manager.cc:1428] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.769968 29705 tablet_bootstrap.cc:654] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.769632 29728 ts_tablet_manager.cc:1939] T 247cf54f5ca04006b41b10459ed6b561 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting consensus metadata
I20260928 02:43:24.770382 29185 catalog_manager.cc:4841] TS b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237): tablet 247cf54f5ca04006b41b10459ed6b561 (table test-table [id=702634611e2e4f9ebba074271df5640d]) successfully deleted
I20260928 02:43:24.770498 29704 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Bootstrap starting.
I20260928 02:43:24.770740 29729 ts_tablet_manager.cc:1905] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:24.770893 29704 tablet_bootstrap.cc:654] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.771484 29704 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: No bootstrap required, opened a new log
I20260928 02:43:24.771543 29704 ts_tablet_manager.cc:1397] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.771685 29704 raft_consensus.cc:348] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.771755 29704 raft_consensus.cc:374] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.771782 29704 raft_consensus.cc:729] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607f4c5f7a0a4cb4a78b333fc1a04a8d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.771822 29704 consensus_queue.cc:260] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.771932 29704 ts_tablet_manager.cc:1428] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:24.772086 29729 ts_tablet_manager.cc:1918] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:24.772153 29729 log.cc:1196] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/247cf54f5ca04006b41b10459ed6b561
I20260928 02:43:24.772410 29729 ts_tablet_manager.cc:1939] T 247cf54f5ca04006b41b10459ed6b561 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting consensus metadata
I20260928 02:43:24.772783 29185 catalog_manager.cc:4841] TS 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959): tablet 247cf54f5ca04006b41b10459ed6b561 (table test-table [id=702634611e2e4f9ebba074271df5640d]) successfully deleted
I20260928 02:43:24.774585 29727 tablet_replica.cc:330] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: stopping tablet replica
I20260928 02:43:24.774693 29727 raft_consensus.cc:2229] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:24.774828 29727 raft_consensus.cc:2258] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:24.775166 29727 ts_tablet_manager.cc:1905] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:24.775341 29705 tablet_bootstrap.cc:492] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: No bootstrap required, opened a new log
I20260928 02:43:24.775401 29705 ts_tablet_manager.cc:1397] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 02:43:24.775556 29705 raft_consensus.cc:348] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.775679 29705 raft_consensus.cc:374] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.775707 29705 raft_consensus.cc:729] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f0b0bd292a4fae85088c28f52fa89d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.775754 29705 consensus_queue.cc:260] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.775846 29705 ts_tablet_manager.cc:1428] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:24.776644 29727 ts_tablet_manager.cc:1918] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:24.776708 29727 log.cc:1196] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/247cf54f5ca04006b41b10459ed6b561
I20260928 02:43:24.776950 29727 ts_tablet_manager.cc:1939] T 247cf54f5ca04006b41b10459ed6b561 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting consensus metadata
I20260928 02:43:24.777333 29183 catalog_manager.cc:4841] TS 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130:38923): tablet 247cf54f5ca04006b41b10459ed6b561 (table test-table [id=702634611e2e4f9ebba074271df5640d]) successfully deleted
I20260928 02:43:24.833848 29722 raft_consensus.cc:482] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:24.833932 29722 raft_consensus.cc:504] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.834080 29722 leader_election.cc:288] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237), 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959)
I20260928 02:43:24.834316 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf09d41f9064322a9a3432d0f6bce45" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" is_pre_election: true
I20260928 02:43:24.834306 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf09d41f9064322a9a3432d0f6bce45" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" is_pre_election: true
I20260928 02:43:24.834404 29355 raft_consensus.cc:2454] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 0.
I20260928 02:43:24.834404 29635 raft_consensus.cc:2454] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 0.
I20260928 02:43:24.834571 29429 leader_election.cc:302] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, 66f0b0bd292a4fae85088c28f52fa89d; no voters:
I20260928 02:43:24.834707 29722 raft_consensus.cc:2781] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:24.834759 29722 raft_consensus.cc:482] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:24.834784 29722 raft_consensus.cc:3037] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.835602 29722 raft_consensus.cc:504] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.835754 29722 leader_election.cc:288] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 election: Requested vote from peers b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237), 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959)
I20260928 02:43:24.835924 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf09d41f9064322a9a3432d0f6bce45" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
I20260928 02:43:24.835935 29635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf09d41f9064322a9a3432d0f6bce45" candidate_uuid: "66f0b0bd292a4fae85088c28f52fa89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d"
I20260928 02:43:24.836014 29355 raft_consensus.cc:3037] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.836030 29635 raft_consensus.cc:3037] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.836651 29635 raft_consensus.cc:2454] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 1.
I20260928 02:43:24.836802 29355 raft_consensus.cc:2454] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66f0b0bd292a4fae85088c28f52fa89d in term 1.
I20260928 02:43:24.836834 29429 leader_election.cc:302] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, 66f0b0bd292a4fae85088c28f52fa89d; no voters:
I20260928 02:43:24.836969 29722 raft_consensus.cc:2781] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:24.837023 29722 raft_consensus.cc:686] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 LEADER]: Becoming Leader. State: Replica: 66f0b0bd292a4fae85088c28f52fa89d, State: Running, Role: LEADER
I20260928 02:43:24.837096 29722 consensus_queue.cc:237] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.837610 29198 catalog_manager.cc:5495] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d reported cstate change: term changed from 0 to 1, leader changed from <none> to 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130). New cstate: current_term: 1 leader_uuid: "66f0b0bd292a4fae85088c28f52fa89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } health_report { overall_health: UNKNOWN } } }
W20260928 02:43:24.857759 29402 tablet.cc:2369] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:43:24 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:24.878367 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.871169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:41213 (local address 127.0.0.1:52798)
0928 02:43:24.871315 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.871334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.871392 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.871538 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.871539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.871836 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:24.871884 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.871885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.872806 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.872808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.873403 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.873404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.873845 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.873846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.873886 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.874419 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:24.874423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:24.875795 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:24.877427 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.877433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.877435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.877614 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.877818 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:24.877819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:24.877819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:24.877871 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:24.878100 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:24.878105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:24.878143 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.878196 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.878278 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260928 02:43:24.878444 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.871231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.27.109.190:41213)
0928 02:43:24.871318 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.871321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.871393 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:43:24.871460 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.871461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.871472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.871526 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:24.871953 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.872729 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.873482 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.873778 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.873841 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.875890 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:24.875891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:24.875892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:24.875898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:24.877347 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.877689 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.877690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.877691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.877752 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:24.877933 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:24.877933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:24.877934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:24.878047 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:24.878097 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.878234 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.878323 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":54}
I20260928 02:43:24.880359 29198 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:52798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:43:24.880597 29198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:24.882023 29475 tablet_service.cc:1467] Processing CreateTablet for tablet b70195535d3c4073b44ff6bc543af70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28d6f21f32904c65937005c6f29e6442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:24.882176 29475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70195535d3c4073b44ff6bc543af70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.883158 29335 tablet_service.cc:1467] Processing CreateTablet for tablet b70195535d3c4073b44ff6bc543af70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28d6f21f32904c65937005c6f29e6442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:24.883168 29615 tablet_service.cc:1467] Processing CreateTablet for tablet b70195535d3c4073b44ff6bc543af70d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28d6f21f32904c65937005c6f29e6442]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:24.883312 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70195535d3c4073b44ff6bc543af70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.883322 29615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70195535d3c4073b44ff6bc543af70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:24.883513 29705 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: Bootstrap starting.
I20260928 02:43:24.883807 29705 tablet_bootstrap.cc:654] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.884378 29705 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: No bootstrap required, opened a new log
I20260928 02:43:24.884430 29705 ts_tablet_manager.cc:1397] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.884546 29705 raft_consensus.cc:348] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.884625 29705 raft_consensus.cc:374] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.884651 29705 raft_consensus.cc:729] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f0b0bd292a4fae85088c28f52fa89d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.884707 29705 consensus_queue.cc:260] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.884797 29705 ts_tablet_manager.cc:1428] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:24.884785 29708 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: Bootstrap starting.
I20260928 02:43:24.884898 29722 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:24.884972 29722 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:24.885066 29704 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Bootstrap starting.
I20260928 02:43:24.885175 29708 tablet_bootstrap.cc:654] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.885485 29704 tablet_bootstrap.cc:654] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:24.885819 29708 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: No bootstrap required, opened a new log
I20260928 02:43:24.885890 29708 ts_tablet_manager.cc:1397] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.886039 29708 raft_consensus.cc:348] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.886119 29704 tablet_bootstrap.cc:492] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: No bootstrap required, opened a new log
I20260928 02:43:24.886126 29708 raft_consensus.cc:374] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.886152 29708 raft_consensus.cc:729] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64a1b4de4334ec2ad7db0c5b6085054, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.886175 29704 ts_tablet_manager.cc:1397] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:24.886202 29708 consensus_queue.cc:260] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.886298 29708 ts_tablet_manager.cc:1428] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.886327 29704 raft_consensus.cc:348] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.886404 29704 raft_consensus.cc:374] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:24.886359 29712 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:24.886423 29704 raft_consensus.cc:729] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607f4c5f7a0a4cb4a78b333fc1a04a8d, State: Initialized, Role: FOLLOWER
I20260928 02:43:24.886443 29712 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:24.886492 29704 consensus_queue.cc:260] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.886581 29704 ts_tablet_manager.cc:1428] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:24.886719 29710 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:24.886821 29710 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:43:24.936029 29682 tablet.cc:2369] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:24.980116 29710 raft_consensus.cc:482] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:24.980206 29710 raft_consensus.cc:504] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.980497 29710 leader_election.cc:288] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237), 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130:38923)
I20260928 02:43:24.984087 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.980672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38923 (local address 127.27.109.131:43369)
0928 02:43:24.980729 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.980742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.980784 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.980959 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.980960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.980961 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.981034 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.981036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.981936 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.981936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.983275 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.983278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.983900 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.983900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.983958 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.983961 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.984002 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.984032 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":10}
I20260928 02:43:24.984133 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.980766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43369 (local address 127.27.109.130:38923)
0928 02:43:24.980817 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.980819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.980828 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:24.980856 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.980857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.980875 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.980954 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.981112 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.981857 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.983357 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.983820 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.983891 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.983899 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.984025 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.984062 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:43:24.984423 29491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70195535d3c4073b44ff6bc543af70d" candidate_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f0b0bd292a4fae85088c28f52fa89d" is_pre_election: true
I20260928 02:43:24.984494 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.980758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43083 (local address 127.27.109.129:40237)
0928 02:43:24.980817 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:24.980819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:24.980937 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:43:24.981031 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:24.981032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:24.981039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:24.981098 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:24.981254 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.981953 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.983582 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.984105 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.984170 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.984178 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:24.984387 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:43:24.984437 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:24.984551 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:24.980692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40237 (local address 127.27.109.131:43083)
0928 02:43:24.980858 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:24.980866 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:24.980941 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:24.981102 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:24.981103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:24.981104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:24.981179 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.981180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.982028 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.982029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.983509 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:24.983511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:24.984214 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:24.984215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:24.984302 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:24.984306 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:24.984358 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:43:24.984483 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260928 02:43:24.984589 29491 raft_consensus.cc:2454] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607f4c5f7a0a4cb4a78b333fc1a04a8d in term 0.
I20260928 02:43:24.984817 29567 leader_election.cc:302] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, 66f0b0bd292a4fae85088c28f52fa89d; no voters:
I20260928 02:43:24.984817 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70195535d3c4073b44ff6bc543af70d" candidate_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" is_pre_election: true
I20260928 02:43:24.984901 29355 raft_consensus.cc:2454] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607f4c5f7a0a4cb4a78b333fc1a04a8d in term 0.
I20260928 02:43:24.984967 29710 raft_consensus.cc:2781] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:24.985011 29710 raft_consensus.cc:482] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:24.985036 29710 raft_consensus.cc:3037] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.985689 29710 raft_consensus.cc:504] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.985832 29710 leader_election.cc:288] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [CANDIDATE]: Term 1 election: Requested vote from peers b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237), 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130:38923)
I20260928 02:43:24.985992 29355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70195535d3c4073b44ff6bc543af70d" candidate_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b64a1b4de4334ec2ad7db0c5b6085054"
I20260928 02:43:24.985988 29491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70195535d3c4073b44ff6bc543af70d" candidate_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f0b0bd292a4fae85088c28f52fa89d"
I20260928 02:43:24.986064 29491 raft_consensus.cc:3037] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.986064 29355 raft_consensus.cc:3037] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:24.986793 29355 raft_consensus.cc:2454] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607f4c5f7a0a4cb4a78b333fc1a04a8d in term 1.
I20260928 02:43:24.986956 29491 raft_consensus.cc:2454] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607f4c5f7a0a4cb4a78b333fc1a04a8d in term 1.
I20260928 02:43:24.986999 29569 leader_election.cc:302] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607f4c5f7a0a4cb4a78b333fc1a04a8d, b64a1b4de4334ec2ad7db0c5b6085054; no voters:
I20260928 02:43:24.987071 29710 raft_consensus.cc:2781] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:24.987244 29710 raft_consensus.cc:686] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 LEADER]: Becoming Leader. State: Replica: 607f4c5f7a0a4cb4a78b333fc1a04a8d, State: Running, Role: LEADER
I20260928 02:43:24.987360 29710 consensus_queue.cc:237] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } }
I20260928 02:43:24.987742 29744 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: TxnStatusTablet state changed. Reason: New leader 607f4c5f7a0a4cb4a78b333fc1a04a8d. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:24.987849 29744 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:24.988024 29746 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:43:24.988135 29746 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:43:24.988129 29198 catalog_manager.cc:5495] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131). New cstate: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } health_report { overall_health: HEALTHY } } }
I20260928 02:43:25.002991 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.000295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39959 (local address 127.0.0.1:45356)
0928 02:43:25.000359 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:25.000372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:25.000424 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:25.000563 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:25.000564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:25.000566 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:25.000612 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.000614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.001203 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.001204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.002125 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.002127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.002449 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.002450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.002531 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.002547 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.002848 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.002857 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:43:25.002898 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:43:25.002935 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:43:25.003013 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.000364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.27.109.131:39959)
0928 02:43:25.000413 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:25.000415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:25.000423 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:25.000493 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:25.000494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:25.000506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:25.000559 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:25.000679 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.001131 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.002205 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.002379 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.002426 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.002670 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.002782 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.002852 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:25.002924 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:43:25.002945 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:43:25.003986 29355 raft_consensus.cc:1261] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Refusing update from remote peer 607f4c5f7a0a4cb4a78b333fc1a04a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:25.004073 29491 raft_consensus.cc:1261] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Refusing update from remote peer 607f4c5f7a0a4cb4a78b333fc1a04a8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:25.004369 29744 consensus_queue.cc:1035] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:25.004505 29710 consensus_queue.cc:1035] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:25.005023 29722 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: TxnStatusTablet state changed. Reason: New leader 607f4c5f7a0a4cb4a78b333fc1a04a8d. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.005111 29722 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:25.005519 29712 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: TxnStatusTablet state changed. Reason: New leader 607f4c5f7a0a4cb4a78b333fc1a04a8d. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.005609 29712 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:25.005961 29718 mvcc.cc:205] Tried to move back new op lower bound from 7334147706894475264 to 7334147706828832768. Current Snapshot: MvccSnapshot[applied={T|T < 7334147706894475264}]
I20260928 02:43:25.006031 29710 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.006098 29710 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:25.006430 29710 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.006521 29710 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 607f4c5f7a0a4cb4a78b333fc1a04a8d: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:25.006605 29712 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.006680 29712 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P b64a1b4de4334ec2ad7db0c5b6085054: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:25.007283 29722 tablet_replica.cc:439] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 } } peers { permanent_uuid: "66f0b0bd292a4fae85088c28f52fa89d" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38923 } } peers { permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 } } }
I20260928 02:43:25.007364 29722 tablet_replica.cc:442] T b70195535d3c4073b44ff6bc543af70d P 66f0b0bd292a4fae85088c28f52fa89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:25.012596 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.010115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39959 (local address 127.27.109.130:42027)
0928 02:43:25.010185 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:25.010196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:25.010242 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:25.010369 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:25.010370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:25.010371 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:25.010407 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.010408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.011271 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.011271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.011818 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.011819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.012267 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.012267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.012304 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.012315 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.012479 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.012480 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:25.012513 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 02:43:25.012548 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 02:43:25.012598 29700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.010200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42027 (local address 127.27.109.131:39959)
0928 02:43:25.010245 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:25.010247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:25.010255 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:25.010296 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:25.010297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:25.010306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:25.010357 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:25.010487 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.011195 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.011897 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.012192 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.012260 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.012390 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.012420 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.012471 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:25.012517 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:43:25.012532 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:43:25.014402 29355 raft_consensus.cc:1261] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Refusing update from remote peer 66f0b0bd292a4fae85088c28f52fa89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:25.014413 29635 raft_consensus.cc:1261] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Refusing update from remote peer 66f0b0bd292a4fae85088c28f52fa89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:25.014635 29722 consensus_queue.cc:1035] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b64a1b4de4334ec2ad7db0c5b6085054" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:25.014758 29709 consensus_queue.cc:1035] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "607f4c5f7a0a4cb4a78b333fc1a04a8d" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:25.029016 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.026320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38923 (local address 127.27.109.131:45573)
0928 02:43:25.026374 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:25.026386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:25.026424 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:25.026557 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:25.026558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:25.026559 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:25.026613 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.026614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.027496 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.027497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.028174 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:25.028176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.028653 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.028654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:25.028699 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.028735 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.028907 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.028908 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:25.028937 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 02:43:25.028967 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 02:43:25.029032 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.026419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:45573 (local address 127.27.109.130:38923)
0928 02:43:25.026455 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:25.026457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:25.026482 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:43:25.026501 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:25.026502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:25.026510 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:25.026556 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:25.026680 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.027426 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.028283 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.028581 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.028644 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.028815 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:25.028848 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:25.028906 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:25.028940 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 02:43:25.028960 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:25.034812 29197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:25.034911 29197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:25.035461 29197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f81d731cf7346e6aa9c0ccd7e806d79: Sending DeleteTablet for 3 replicas of tablet faf09d41f9064322a9a3432d0f6bce45
I20260928 02:43:25.035703 29475 tablet_service.cc:1514] Processing DeleteTablet for tablet faf09d41f9064322a9a3432d0f6bce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:25 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:59266
I20260928 02:43:25.035717 29335 tablet_service.cc:1514] Processing DeleteTablet for tablet faf09d41f9064322a9a3432d0f6bce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:25 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 02:43:25.035802 29727 tablet_replica.cc:330] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: stopping tablet replica
I20260928 02:43:25.035804 29728 tablet_replica.cc:330] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: stopping tablet replica
I20260928 02:43:25.035847 29728 raft_consensus.cc:2229] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:25.035849 29727 raft_consensus.cc:2229] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:25.035851 29615 tablet_service.cc:1514] Processing DeleteTablet for tablet faf09d41f9064322a9a3432d0f6bce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:25 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45348
I20260928 02:43:25.035882 29728 raft_consensus.cc:2258] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:25.035893 29727 raft_consensus.cc:2258] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:25.035921 29729 tablet_replica.cc:330] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: stopping tablet replica
I20260928 02:43:25.035964 29729 raft_consensus.cc:2229] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:25.036010 29729 raft_consensus.cc:2258] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:25.036198 29728 ts_tablet_manager.cc:1905] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:25.036348 29729 ts_tablet_manager.cc:1905] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:25.037858 29728 ts_tablet_manager.cc:1918] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:25.037884 29729 ts_tablet_manager.cc:1918] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:25.037915 29728 log.cc:1196] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/faf09d41f9064322a9a3432d0f6bce45
I20260928 02:43:25.037936 29729 log.cc:1196] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/faf09d41f9064322a9a3432d0f6bce45
I20260928 02:43:25.038107 29727 ts_tablet_manager.cc:1905] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:25.038147 29728 ts_tablet_manager.cc:1939] T faf09d41f9064322a9a3432d0f6bce45 P b64a1b4de4334ec2ad7db0c5b6085054: Deleting consensus metadata
I20260928 02:43:25.038177 29729 ts_tablet_manager.cc:1939] T faf09d41f9064322a9a3432d0f6bce45 P 607f4c5f7a0a4cb4a78b333fc1a04a8d: Deleting consensus metadata
I20260928 02:43:25.038489 29185 catalog_manager.cc:4841] TS b64a1b4de4334ec2ad7db0c5b6085054 (127.27.109.129:40237): tablet faf09d41f9064322a9a3432d0f6bce45 (table test-table [id=78bd06a099a5444f8754fd04eb3266a5]) successfully deleted
I20260928 02:43:25.038548 29185 catalog_manager.cc:4841] TS 607f4c5f7a0a4cb4a78b333fc1a04a8d (127.27.109.131:39959): tablet faf09d41f9064322a9a3432d0f6bce45 (table test-table [id=78bd06a099a5444f8754fd04eb3266a5]) successfully deleted
I20260928 02:43:25.039709 29727 ts_tablet_manager.cc:1918] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:25.039748 29727 log.cc:1196] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/faf09d41f9064322a9a3432d0f6bce45
I20260928 02:43:25.039892 29727 ts_tablet_manager.cc:1939] T faf09d41f9064322a9a3432d0f6bce45 P 66f0b0bd292a4fae85088c28f52fa89d: Deleting consensus metadata
I20260928 02:43:25.040266 29183 catalog_manager.cc:4841] TS 66f0b0bd292a4fae85088c28f52fa89d (127.27.109.130:38923): tablet faf09d41f9064322a9a3432d0f6bce45 (table test-table [id=78bd06a099a5444f8754fd04eb3266a5]) successfully deleted
Sep 28 02:43:25 dist-test-slave-b5p5 krb5kdc[29153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563404, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:25.047531 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.037826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.27.109.129:40237)
0928 02:43:25.037863 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:25.037865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:25.038740 (+ 875us) server_negotiation.cc:407] Connection header received
0928 02:43:25.038789 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:25.038789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:25.038800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:25.038858 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:25.039529 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.040315 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.041344 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.041522 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.041574 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.044768 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:25.044776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:25.044776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:25.044783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:25.046348 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:25.046737 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:25.046740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:25.046741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:25.046799 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:25.047010 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:25.047010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:25.047011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:25.047148 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:25.047295 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:25.047365 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:43:25.047436 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:43:25.047835 29395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59344
I20260928 02:43:25.055578 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:25.048754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52802 (local address 127.27.109.190:41213)
0928 02:43:25.048801 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:25.048803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:25.048925 (+ 122us) server_negotiation.cc:407] Connection header received
0928 02:43:25.049007 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:25.049008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:25.049017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:25.049061 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:25.049654 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.050238 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.051248 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:25.051537 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:25.051576 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:25.052992 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:25.052994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:25.052994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:25.053001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:25.054591 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:25.054896 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:25.054897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:25.054898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:25.054956 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:25.055124 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:25.055125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:25.055125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:25.055248 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:25.055341 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:25.055413 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:43:25.055491 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 02:43:25.055826 29197 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52802
I20260928 02:43:25.056561 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29270
I20260928 02:43:25.063696 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29410
I20260928 02:43:25.070742 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29550
I20260928 02:43:25.075435 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29166
2026-09-28T02:43:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2189 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:25 dist-test-slave-b5p5 krb5kdc[29765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:25 dist-test-slave-b5p5 krb5kdc[29765](info): set up 2 sockets
Sep 28 02:43:25 dist-test-slave-b5p5 krb5kdc[29765](info): commencing operation
krb5kdc: starting...
W20260928 02:43:26.110543 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:26.220986 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:33657
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:26.302166 29778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:26.302413 29778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:26.302462 29778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:26.303591 29778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:26.303658 29778 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:43:26.303683 29778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:26.303716 29778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:26.303740 29778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:26.303767 29778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:26.305177 29778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:33657
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:26.305428 29778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:26.305635 29778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:26.310231 29786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.310231 29783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.310264 29784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:26.310489 29778 server_base.cc:1034] running on GCE node
I20260928 02:43:26.310735 29778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:26.310995 29778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:26.312129 29778 hybrid_clock.cc:648] HybridClock initialized: now 1790563406312108 us; error 36 us; skew 500 ppm
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:26.314592 29778 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:26.315017 29778 webserver.cc:460] Webserver started at http://127.27.109.190:37647/ using document root <none> and password file <none>
I20260928 02:43:26.315228 29778 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:26.315271 29778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:26.315407 29778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:26.316360 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "0c759d6a1aca46c6b8e64d25d012bcbd"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "310139844417867e88ab8653dd17804b"
server_key_iv: "8647c4b4a09555a812052b5fb2228c4b"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.316704 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "0c759d6a1aca46c6b8e64d25d012bcbd"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "310139844417867e88ab8653dd17804b"
server_key_iv: "8647c4b4a09555a812052b5fb2228c4b"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.317952 29778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:26.318769 29793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.318923 29778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:26.319007 29778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "0c759d6a1aca46c6b8e64d25d012bcbd"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "310139844417867e88ab8653dd17804b"
server_key_iv: "8647c4b4a09555a812052b5fb2228c4b"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.319065 29778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:26.339112 29778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:26.341274 29778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:26.341393 29778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:26.346129 29778 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:33657
I20260928 02:43:26.346225 29855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:33657 every 8 connection(s)
I20260928 02:43:26.346587 29778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:26.347285 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.349376 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Bootstrap starting.
I20260928 02:43:26.349887 29856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.350108 29856 log.cc:824] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:26.350847 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: No bootstrap required, opened a new log
I20260928 02:43:26.352823 29856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } }
I20260928 02:43:26.352953 29856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.352974 29856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c759d6a1aca46c6b8e64d25d012bcbd, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.353103 29856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } }
I20260928 02:43:26.353183 29856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:26.353204 29856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:26.353231 29856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:26.353703 29856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } }
I20260928 02:43:26.353785 29856 leader_election.cc:302] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c759d6a1aca46c6b8e64d25d012bcbd; no voters:
I20260928 02:43:26.353929 29856 leader_election.cc:288] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:26.353952 29859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:26.354066 29859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [term 1 LEADER]: Becoming Leader. State: Replica: 0c759d6a1aca46c6b8e64d25d012bcbd, State: Running, Role: LEADER
I20260928 02:43:26.354182 29856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:26.354218 29859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } }
I20260928 02:43:26.354625 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } } }
I20260928 02:43:26.354658 29861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c759d6a1aca46c6b8e64d25d012bcbd. Latest consensus state: current_term: 1 leader_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c759d6a1aca46c6b8e64d25d012bcbd" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 33657 } } }
I20260928 02:43:26.354789 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:26.354810 29861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:26.355225 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29778
I20260928 02:43:26.355291 29868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:26.355322 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:26.355499 28086 external_mini_cluster.cc:1428] Setting key 1b2b13ae6e3dac54a281ac79f73daa61
I20260928 02:43:26.355836 29868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:26.357481 29868 catalog_manager.cc:1347] Generated new cluster ID: e31fe793e64d4bc885c86e237c3c940b
I20260928 02:43:26.357539 29868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.366989 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.356115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.27.109.190:33657)
0928 02:43:26.356295 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.356306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.356335 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:43:26.356520 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.356528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.356583 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.356721 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.357488 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.358875 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.359953 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.360554 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.360643 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.363423 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.363428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.363428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.363440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.365399 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.365763 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.365766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.365766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.365878 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.366077 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.366078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.366078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.366372 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.366572 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.366649 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.366783 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260928 02:43:26.377849 29868 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:26.378597 29868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:26.386291 29868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Generated new TSK 0
I20260928 02:43:26.386449 29868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:26.412993 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:33657
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:26.491277 29882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:26.491462 29882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:26.491485 29882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:26.492534 29882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:26.492569 29882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:26.492589 29882 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:43:26.492601 29882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:26.492619 29882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:26.493988 29882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:26.494167 29882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:26.494331 29882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:26.499418 29888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.499409 29890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.499408 29887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:26.499459 29882 server_base.cc:1034] running on GCE node
I20260928 02:43:26.499752 29882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:26.499934 29882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:26.501066 29882 hybrid_clock.cc:648] HybridClock initialized: now 1790563406501055 us; error 27 us; skew 500 ppm
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:26.503443 29882 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:26.503842 29882 webserver.cc:460] Webserver started at http://127.27.109.129:44915/ using document root <none> and password file <none>
I20260928 02:43:26.504069 29882 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:26.504143 29882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:26.504288 29882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:26.505273 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "47eb0b6c29fe3a9a972e3b7c604d762d"
server_key_iv: "009ecf282f3bdf5cb6c9192461614052"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.505617 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "47eb0b6c29fe3a9a972e3b7c604d762d"
server_key_iv: "009ecf282f3bdf5cb6c9192461614052"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.507026 29882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:26.507795 29897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.507961 29882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:26.508060 29882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "47eb0b6c29fe3a9a972e3b7c604d762d"
server_key_iv: "009ecf282f3bdf5cb6c9192461614052"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.508152 29882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:26.518865 29882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:26.520542 29882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:26.520695 29882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:26.521207 29882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:26.521270 29882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.521308 29882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:26.521337 29882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.527879 29882 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:33881
I20260928 02:43:26.527930 30012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:33881 every 8 connection(s)
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.528223 29882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:26.531023 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.521627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:39129 (local address 127.27.109.190:33657)
0928 02:43:26.521680 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.521683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.521916 (+ 233us) server_negotiation.cc:407] Connection header received
0928 02:43:26.522344 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.522346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.522362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.522444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.523268 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.524105 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.525692 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.526058 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.526135 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.528759 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.528760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.528762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.528775 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.529868 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.530202 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.530203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.530203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.530250 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.530417 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.530417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.530418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.530577 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.530680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.530901 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.530964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:43:26.531159 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.521646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.129:39129)
0928 02:43:26.521783 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.521820 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.522256 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.522545 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.522549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.522878 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.523159 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.523165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.524230 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.524231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.525585 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.525588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.526168 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.526168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.526293 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.526825 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.526837 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.528662 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.529933 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.529938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.529940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.530137 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.530290 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.530290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.530291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.530369 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.530690 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.530695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.530795 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.530875 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.530981 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260928 02:43:26.531646 30013 heartbeater.cc:347] Connected to a master server at 127.27.109.190:33657
I20260928 02:43:26.531723 30013 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:26.531891 30013 heartbeater.cc:511] Master 127.27.109.190:33657 requested a full tablet report, sending...
I20260928 02:43:26.532408 29809 ts_manager.cc:194] Registered new tserver with Master: d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881)
I20260928 02:43:26.533303 29809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:39129
I20260928 02:43:26.537165 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 29882
I20260928 02:43:26.537294 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:26.537480 28086 external_mini_cluster.cc:1428] Setting key 6dc1214603d410b0bd0411564a675c07
I20260928 02:43:26.537886 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.532712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:46691 (local address 127.27.109.190:33657)
0928 02:43:26.532780 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.532781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.532808 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:43:26.532871 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.532871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.532878 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.532920 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.533213 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.533797 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.534495 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.534710 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.534794 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.535579 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.535581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.535581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.535596 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.536771 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.537091 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.537092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.537092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.537147 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.537338 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.537339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.537339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.537476 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.537557 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.537735 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.537803 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:43:26.537902 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.532627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.129:46691)
0928 02:43:26.532750 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.532760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.532819 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.532926 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.532926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.533092 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.533154 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.533155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.533890 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.533891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.534400 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.534401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.534795 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.534796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.534876 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.535223 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.535227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.535493 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.536853 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.536854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.536855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.537019 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.537223 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.537224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.537225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.537280 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.537559 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.537559 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.537617 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.537708 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.537803 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:26.561599 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:33657
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:26.639297 30022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:26.639484 30022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:26.639508 30022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:26.640627 30022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:26.640666 30022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:26.640686 30022 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:43:26.640698 30022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:26.640743 30022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:26.642153 30022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:26.642371 30022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:26.642623 30022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:26.647460 30028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.647476 30030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.647472 30027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:26.647787 30022 server_base.cc:1034] running on GCE node
I20260928 02:43:26.647975 30022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:26.648206 30022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:26.649362 30022 hybrid_clock.cc:648] HybridClock initialized: now 1790563406649357 us; error 34 us; skew 500 ppm
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:26.651731 30022 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:26.652123 30022 webserver.cc:460] Webserver started at http://127.27.109.130:33145/ using document root <none> and password file <none>
I20260928 02:43:26.652338 30022 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:26.652402 30022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:26.652551 30022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:26.653563 30022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "f0d1bbc9fa3cafc4c72f729d94aaef90"
server_key_iv: "4639c6ca2742e86824e6a835717e5cd7"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.653920 30022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "f0d1bbc9fa3cafc4c72f729d94aaef90"
server_key_iv: "4639c6ca2742e86824e6a835717e5cd7"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.655375 30022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:26.656229 30037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.656411 30022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:26.656507 30022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "f0d1bbc9fa3cafc4c72f729d94aaef90"
server_key_iv: "4639c6ca2742e86824e6a835717e5cd7"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.656594 30022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:26.675340 30022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:26.677641 30022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:26.677803 30022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:26.678272 30022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:26.678337 30022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.678381 30022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:26.678411 30022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.685570 30022 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:45937
I20260928 02:43:26.685631 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:45937 every 8 connection(s)
I20260928 02:43:26.685985 30022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:26.688136 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.678846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:33199 (local address 127.27.109.190:33657)
0928 02:43:26.678924 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.678927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.679121 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:43:26.679562 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.679564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.679577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.679656 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.680346 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.681140 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.682796 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.683110 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.683185 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.685586 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.685587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.685588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.685594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.687105 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.687449 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.687449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.687450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.687496 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.687670 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.687670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.687671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.687763 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.687812 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.688012 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.688075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:43:26.688292 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.678803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.130:33199)
0928 02:43:26.678973 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.679013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.679471 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.679741 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.679746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.680051 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.680246 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.680251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.681244 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.681246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.682708 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.682711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.683263 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.683265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.683477 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.684078 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.684090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.685479 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.687173 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.687178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.687179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.687385 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.687537 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.687537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.687538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.687621 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.687818 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.687822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.687923 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.688000 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.688105 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 02:43:26.688723 30153 heartbeater.cc:347] Connected to a master server at 127.27.109.190:33657
I20260928 02:43:26.688800 30153 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:26.688979 30153 heartbeater.cc:511] Master 127.27.109.190:33657 requested a full tablet report, sending...
I20260928 02:43:26.689313 29810 ts_manager.cc:194] Registered new tserver with Master: 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937)
I20260928 02:43:26.689832 29810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:33199
I20260928 02:43:26.694794 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.689709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.130:34265)
0928 02:43:26.689821 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.689831 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.689885 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.690122 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.690123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.690258 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.690309 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.690310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.691003 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.691003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.691504 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.691505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.691858 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.691859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.691930 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.692264 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.692268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.692534 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.693742 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.693743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.693744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.693919 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.694110 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.694111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.694111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.694180 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.694448 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.694448 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.694521 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.694601 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.694695 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":69,"threads_started":1}
I20260928 02:43:26.694810 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.689771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34265 (local address 127.27.109.190:33657)
0928 02:43:26.689809 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.689811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.689875 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:43:26.689924 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.689925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.689934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.689994 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.690386 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.690914 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.691583 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.691786 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.691859 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.692625 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.692626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.692626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.692632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.693662 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.693988 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.693989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.693990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.694045 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.694242 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.694243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.694243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.694378 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.694455 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.694663 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.694728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:26.696075 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30022
I20260928 02:43:26.696197 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:26.696365 28086 external_mini_cluster.cc:1428] Setting key dafb91e3d01685eeed0558b7be80c5ba
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:26.719684 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:33657
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:26.799484 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:26.799682 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:26.799705 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:26.800750 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:26.800782 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:26.800802 30162 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:43:26.800814 30162 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:43:26.800830 30162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:26.802150 30162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:26.802316 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:26.802510 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:26.807370 30167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.807384 30168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:26.807376 30170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:26.807668 30162 server_base.cc:1034] running on GCE node
I20260928 02:43:26.807863 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:26.808118 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:26.809274 30162 hybrid_clock.cc:648] HybridClock initialized: now 1790563406809256 us; error 27 us; skew 500 ppm
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:26.811990 30162 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:26.812405 30162 webserver.cc:460] Webserver started at http://127.27.109.131:41375/ using document root <none> and password file <none>
I20260928 02:43:26.812631 30162 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:26.812675 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:26.812806 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:26.813844 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "c09185879593487389a76d0571aab0da"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "eda128cbd84fd65fb6c2b69127253581"
server_key_iv: "2a919dde0a9f9ac02aa382387e3d12f5"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.814186 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "c09185879593487389a76d0571aab0da"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "eda128cbd84fd65fb6c2b69127253581"
server_key_iv: "2a919dde0a9f9ac02aa382387e3d12f5"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.815522 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:26.816262 30177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.816483 30162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:26.816591 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "c09185879593487389a76d0571aab0da"
format_stamp: "Formatted at 2026-09-28 02:43:26 on dist-test-slave-b5p5"
server_key: "eda128cbd84fd65fb6c2b69127253581"
server_key_iv: "2a919dde0a9f9ac02aa382387e3d12f5"
server_key_version: "encryptionkey@0"
I20260928 02:43:26.816666 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:26.837313 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:26.839566 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:26.839731 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:26.840219 30162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:26.840256 30162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.840283 30162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:26.840297 30162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.847256 30162 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:34081
I20260928 02:43:26.847311 30292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:34081 every 8 connection(s)
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.847627 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:26.850616 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.840719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:45401 (local address 127.27.109.190:33657)
0928 02:43:26.840790 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.840794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.840987 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:43:26.841467 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.841469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.841482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.841558 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.842458 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.843296 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.844882 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.845199 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.845263 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.847860 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.847862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.847862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.847870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.849250 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.849655 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.849656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.849657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.849715 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.849959 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.849960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.849960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.850079 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.850155 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.850411 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.850511 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:26.850837 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.840690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.131:45401)
0928 02:43:26.840861 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.840901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.841373 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.841691 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.841696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.842081 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.842354 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.842360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.843402 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.843403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.844799 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.844801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.845287 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.845288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.845443 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.845984 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.845994 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.847771 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.849349 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.849355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.849357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.849570 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.849787 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.849788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.849788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.849889 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.850146 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.850151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.850283 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.850390 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.850615 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
I20260928 02:43:26.851290 30293 heartbeater.cc:347] Connected to a master server at 127.27.109.190:33657
I20260928 02:43:26.851388 30293 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:26.851562 30293 heartbeater.cc:511] Master 127.27.109.190:33657 requested a full tablet report, sending...
I20260928 02:43:26.851967 29809 ts_manager.cc:194] Registered new tserver with Master: c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:26.852445 29809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:45401
I20260928 02:43:26.854203 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30162
I20260928 02:43:26.854305 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:26.854521 28086 external_mini_cluster.cc:1428] Setting key c78b02e1f265fc759ce89cbb0d0f1fab
I20260928 02:43:26.855372 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:26.858249 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.852294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:53695 (local address 127.27.109.190:33657)
0928 02:43:26.852542 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.852543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.852551 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:26.852592 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.852592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.852601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.852655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.853012 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.853440 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.854172 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.854457 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.854550 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.855583 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.855584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.855585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.855591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.857204 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.857612 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.857613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.857613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.857660 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.857855 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.857855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.857855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.857941 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.857988 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.858149 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.858198 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260928 02:43:26.858376 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.852239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.27.109.131:53695)
0928 02:43:26.852346 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.852356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.852407 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.852676 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.852677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.852876 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.852940 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.852942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.853521 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.853522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.854089 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.854091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.854550 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.854551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.854645 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.855085 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.855091 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.855502 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.857304 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.857305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.857306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.857538 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.857725 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.857726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.857726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.857795 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.858000 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.858001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.858049 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.858135 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.858238 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.877892 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.871511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.27.109.190:33657)
0928 02:43:26.871605 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.871607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.871614 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:26.871633 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.871634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.871641 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.871688 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.872181 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.872662 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.873466 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.873682 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.873738 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.875881 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.875882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.875883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.875888 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.877031 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.877319 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.877320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.877320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.877366 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.877520 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.877520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.877520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.877606 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.877676 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.877787 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.877835 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:43:26.879526 29809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:26.881024 29809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:26.887065 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.882974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:33881 (local address 127.0.0.1:36782)
0928 02:43:26.883125 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.883164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.883222 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.883506 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.883510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.883512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:26.883591 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.883593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.884623 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.884625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.885927 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.885929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886758 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.886840 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886852 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.886922 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.886977 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":37,"thread_start_us":54,"threads_started":1}
I20260928 02:43:26.887132 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.883363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.27.109.130:45937)
0928 02:43:26.883694 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.883698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.883707 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:26.883743 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.883745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.883761 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.883833 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:26.884016 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.885028 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886019 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886762 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886887 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.886975 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.887015 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":75,"threads_started":1}
I20260928 02:43:26.887223 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.883072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.27.109.131:34081)
0928 02:43:26.883238 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.883243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.883413 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:43:26.883488 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.883491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.883510 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.883585 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:26.883701 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.884806 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886063 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886770 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886838 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886887 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.887084 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.887125 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 02:43:26.887068 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.883152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.0.0.1:57666)
0928 02:43:26.883487 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.883495 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.883534 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.883897 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.883898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.883901 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:26.883940 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.883941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.885119 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.885120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.885955 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.885957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886846 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.886900 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886903 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.886961 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.886991 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260928 02:43:26.887478 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.882974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:34081 (local address 127.0.0.1:45874)
0928 02:43:26.883327 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.883336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.883412 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.883595 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.883596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.883597 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:26.883638 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.883640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.884891 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.884891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.885962 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.885964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886922 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.886989 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886994 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.887398 (+ 404us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.887432 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260928 02:43:26.887552 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.883083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.27.109.129:33881)
0928 02:43:26.883264 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.883267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.883277 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:26.883321 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.883323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.883340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.883430 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:26.883676 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.884528 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886031 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.886589 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.886676 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.886738 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.887423 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.887466 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260928 02:43:26.887991 30087 tablet_service.cc:1467] Processing CreateTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 (DEFAULT_TABLE table=test-table [id=8eb0a0b123734b24816d50227cfee915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:26.888257 30087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f83539d78e46cc98ab1ecb8d2c5432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.888187 29947 tablet_service.cc:1467] Processing CreateTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 (DEFAULT_TABLE table=test-table [id=8eb0a0b123734b24816d50227cfee915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:26.888527 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f83539d78e46cc98ab1ecb8d2c5432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.890390 30227 tablet_service.cc:1467] Processing CreateTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 (DEFAULT_TABLE table=test-table [id=8eb0a0b123734b24816d50227cfee915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:26.890779 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f83539d78e46cc98ab1ecb8d2c5432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.891160 30313 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Bootstrap starting.
I20260928 02:43:26.891793 30313 tablet_bootstrap.cc:654] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.891790 30314 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Bootstrap starting.
I20260928 02:43:26.892210 30313 log.cc:824] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:26.892400 30314 tablet_bootstrap.cc:654] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.892735 30314 log.cc:824] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:26.892953 30313 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: No bootstrap required, opened a new log
I20260928 02:43:26.893024 30313 ts_tablet_manager.cc:1397] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:26.893371 30314 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: No bootstrap required, opened a new log
I20260928 02:43:26.893448 30314 ts_tablet_manager.cc:1397] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:26.894120 30317 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Bootstrap starting.
I20260928 02:43:26.894873 30313 raft_consensus.cc:348] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.895022 30313 raft_consensus.cc:374] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.895036 30317 tablet_bootstrap.cc:654] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.895063 30313 raft_consensus.cc:729] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.895172 30313 consensus_queue.cc:260] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.895356 30317 log.cc:824] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:26.895447 30313 ts_tablet_manager.cc:1428] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:43:26.895507 30153 heartbeater.cc:503] Master 127.27.109.190:33657 was elected leader, sending a full tablet report...
I20260928 02:43:26.895552 30314 raft_consensus.cc:348] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.895691 30314 raft_consensus.cc:374] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.895740 30314 raft_consensus.cc:729] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.895861 30314 consensus_queue.cc:260] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.896135 30314 ts_tablet_manager.cc:1428] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:26.896155 30013 heartbeater.cc:503] Master 127.27.109.190:33657 was elected leader, sending a full tablet report...
I20260928 02:43:26.896935 30317 tablet_bootstrap.cc:492] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: No bootstrap required, opened a new log
I20260928 02:43:26.897004 30317 ts_tablet_manager.cc:1397] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:43:26.898538 30317 raft_consensus.cc:348] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.898664 30317 raft_consensus.cc:374] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.898686 30317 raft_consensus.cc:729] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09185879593487389a76d0571aab0da, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.898808 30317 consensus_queue.cc:260] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.899078 30317 ts_tablet_manager.cc:1428] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:26.899080 30293 heartbeater.cc:503] Master 127.27.109.190:33657 was elected leader, sending a full tablet report...
I20260928 02:43:26.933365 30321 raft_consensus.cc:482] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:26.933463 30321 raft_consensus.cc:504] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.933828 30321 leader_election.cc:288] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937)
W20260928 02:43:26.936540 30154 tablet.cc:2369] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:26.936691 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.934120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:59107 (local address 127.27.109.129:33881)
0928 02:43:26.934180 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.934182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.934191 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:26.934257 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.934258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.934269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.934334 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:26.934556 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.935030 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.935976 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.936334 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.936428 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.936436 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.936596 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.936631 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:43:26.936691 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.934028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:33881 (local address 127.27.109.131:59107)
0928 02:43:26.934093 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.934105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.934177 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.934379 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.934380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.934382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:26.934439 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.934440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.935165 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.935166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.935879 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.935881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.936427 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.936428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.936511 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.936514 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.936578 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.936622 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:43:26.937075 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.934160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.27.109.131:49833)
0928 02:43:26.934331 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.934340 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.934414 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.934554 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.934555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.934556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:26.934627 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.934629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.935224 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.935225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.935921 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.935922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.936805 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.936807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.936890 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.936893 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.936953 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.936987 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260928 02:43:26.937054 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f83539d78e46cc98ab1ecb8d2c5432" candidate_uuid: "c09185879593487389a76d0571aab0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" is_pre_election: true
I20260928 02:43:26.937192 29967 raft_consensus.cc:2454] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09185879593487389a76d0571aab0da in term 0.
I20260928 02:43:26.937431 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.934252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:49833 (local address 127.27.109.130:45937)
0928 02:43:26.934295 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.934297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.934405 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:43:26.934484 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.934485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.934493 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.934549 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:26.934688 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.935135 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.936010 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.936368 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.936455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.936462 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.937322 (+ 860us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.937375 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:26.937448 30179 leader_election.cc:302] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09185879593487389a76d0571aab0da, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:26.937642 30321 raft_consensus.cc:2781] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:26.937714 30321 raft_consensus.cc:482] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:26.937739 30321 raft_consensus.cc:3037] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:26.937704 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f83539d78e46cc98ab1ecb8d2c5432" candidate_uuid: "c09185879593487389a76d0571aab0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" is_pre_election: true
I20260928 02:43:26.937821 30107 raft_consensus.cc:2454] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09185879593487389a76d0571aab0da in term 0.
I20260928 02:43:26.938555 30321 raft_consensus.cc:504] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.938655 30321 leader_election.cc:288] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [CANDIDATE]: Term 1 election: Requested vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937)
I20260928 02:43:26.938916 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f83539d78e46cc98ab1ecb8d2c5432" candidate_uuid: "c09185879593487389a76d0571aab0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6"
I20260928 02:43:26.938920 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f83539d78e46cc98ab1ecb8d2c5432" candidate_uuid: "c09185879593487389a76d0571aab0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
I20260928 02:43:26.938997 30107 raft_consensus.cc:3037] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:26.938997 29967 raft_consensus.cc:3037] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:26.939845 29967 raft_consensus.cc:2454] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09185879593487389a76d0571aab0da in term 1.
I20260928 02:43:26.939845 30107 raft_consensus.cc:2454] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09185879593487389a76d0571aab0da in term 1.
I20260928 02:43:26.940007 30179 leader_election.cc:302] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, c09185879593487389a76d0571aab0da; no voters:
I20260928 02:43:26.940109 30321 raft_consensus.cc:2781] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:26.940243 30321 raft_consensus.cc:686] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 1 LEADER]: Becoming Leader. State: Replica: c09185879593487389a76d0571aab0da, State: Running, Role: LEADER
I20260928 02:43:26.940347 30321 consensus_queue.cc:237] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } }
I20260928 02:43:26.941090 29810 catalog_manager.cc:5495] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da reported cstate change: term changed from 0 to 1, leader changed from <none> to c09185879593487389a76d0571aab0da (127.27.109.131). New cstate: current_term: 1 leader_uuid: "c09185879593487389a76d0571aab0da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:43:26 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:26.974154 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.967138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.27.109.190:33657)
0928 02:43:26.967205 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:26.967209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:26.967339 (+ 130us) server_negotiation.cc:407] Connection header received
0928 02:43:26.967367 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:26.967367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:26.967379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:26.967452 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:26.967862 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.968330 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.968963 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.969137 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.969188 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.971321 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:26.971322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:26.971323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:26.971330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:26.972741 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.973367 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.973368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.973368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.973426 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:26.973631 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:26.973633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:26.973633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:26.973738 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:26.973814 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:26.973959 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:43:26.974040 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:26.974236 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:26.967075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:33657 (local address 127.0.0.1:36396)
0928 02:43:26.967232 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:26.967254 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:26.967309 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:26.967461 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:26.967461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:26.967748 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:26.967789 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.967791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.968396 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.968396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.968899 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:26.968900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:26.969199 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:26.969199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:26.969238 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:26.969766 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:26.969771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:26.971238 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:26.973031 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.973039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.973041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.973273 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.973491 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:26.973492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:26.973492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:26.973549 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:26.973803 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:26.973809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:26.973852 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:43:26.973939 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:43:26.974109 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":77,"thread_start_us":102,"threads_started":1}
I20260928 02:43:26.976176 29810 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:36396:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:43:26.976405 29810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:26.977747 30227 tablet_service.cc:1467] Processing CreateTablet for tablet 2587aded043a4d3fa707c0528bbedfcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54b2b2434d6f480db855d7c4c6d1ae8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:26.977761 29947 tablet_service.cc:1467] Processing CreateTablet for tablet 2587aded043a4d3fa707c0528bbedfcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54b2b2434d6f480db855d7c4c6d1ae8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:26.977766 30087 tablet_service.cc:1467] Processing CreateTablet for tablet 2587aded043a4d3fa707c0528bbedfcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54b2b2434d6f480db855d7c4c6d1ae8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:43:26.977912 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587aded043a4d3fa707c0528bbedfcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.977912 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587aded043a4d3fa707c0528bbedfcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.977936 30087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2587aded043a4d3fa707c0528bbedfcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:26.979189 30313 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Bootstrap starting.
I20260928 02:43:26.979468 30314 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: Bootstrap starting.
I20260928 02:43:26.979854 30313 tablet_bootstrap.cc:654] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.979857 30317 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: Bootstrap starting.
I20260928 02:43:26.980185 30314 tablet_bootstrap.cc:654] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.980252 30317 tablet_bootstrap.cc:654] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:26.980919 30317 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: No bootstrap required, opened a new log
I20260928 02:43:26.980919 30313 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: No bootstrap required, opened a new log
I20260928 02:43:26.980988 30317 ts_tablet_manager.cc:1397] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:26.980983 30313 ts_tablet_manager.cc:1397] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:26.981169 30317 raft_consensus.cc:348] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.981159 30313 raft_consensus.cc:348] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.981251 30317 raft_consensus.cc:374] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.981266 30313 raft_consensus.cc:374] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.981277 30317 raft_consensus.cc:729] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09185879593487389a76d0571aab0da, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.981294 30313 raft_consensus.cc:729] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.981320 30317 consensus_queue.cc:260] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.981344 30313 consensus_queue.cc:260] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.981427 30317 ts_tablet_manager.cc:1428] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:26.981448 30313 ts_tablet_manager.cc:1428] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:26.981510 30318 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:26.981563 30321 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:26.981608 30318 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:26.981643 30321 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:26.981848 30314 tablet_bootstrap.cc:492] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: No bootstrap required, opened a new log
I20260928 02:43:26.981894 30314 ts_tablet_manager.cc:1397] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:26.982036 30314 raft_consensus.cc:348] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.982115 30314 raft_consensus.cc:374] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:26.982133 30314 raft_consensus.cc:729] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Initialized, Role: FOLLOWER
I20260928 02:43:26.982187 30314 consensus_queue.cc:260] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:26.982283 30314 ts_tablet_manager.cc:1428] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:26.982364 30320 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:26.982479 30320 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:43:27.028944 30014 tablet.cc:2369] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:27.098317 30294 tablet.cc:2369] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:27.137017 30318 raft_consensus.cc:482] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:27.137104 30318 raft_consensus.cc:504] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.137449 30318 leader_election.cc:288] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:27.140311 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.137770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:36583 (local address 127.27.109.131:34081)
0928 02:43:27.137830 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.137836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.137898 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:43:27.137951 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.137953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.137962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.138013 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:27.138171 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.138634 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.139564 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.139924 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.140008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.140018 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.140177 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.140213 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:43:27.140312 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.137678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42895 (local address 127.27.109.129:33881)
0928 02:43:27.137732 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.137735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.137745 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:27.137875 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.137876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.137884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.137957 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:27.138124 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.138684 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.139578 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.139926 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.140010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.140018 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.140178 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.140215 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:43:27.140322 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.137620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:33881 (local address 127.27.109.130:42895)
0928 02:43:27.137692 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.137703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.137751 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.137962 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.137963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.137965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:27.138036 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.138038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.138765 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.138766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.139473 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.139475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.140007 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.140008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.140098 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.140101 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.140169 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.140205 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":2}
I20260928 02:43:27.140331 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.137657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:34081 (local address 127.27.109.130:36583)
0928 02:43:27.137818 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.137828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.137902 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.138027 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.138028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.138030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:27.138098 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.138100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.138715 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.138716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.139473 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.139475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.140024 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.140025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.140095 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.140098 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.140160 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.140199 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260928 02:43:27.140616 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587aded043a4d3fa707c0528bbedfcf" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" is_pre_election: true
I20260928 02:43:27.140661 30247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587aded043a4d3fa707c0528bbedfcf" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da" is_pre_election: true
I20260928 02:43:27.140736 29967 raft_consensus.cc:2454] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:27.140795 30247 raft_consensus.cc:2454] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:27.140936 30039 leader_election.cc:302] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:27.141089 30318 raft_consensus.cc:2781] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:27.141141 30318 raft_consensus.cc:482] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:27.141170 30318 raft_consensus.cc:3037] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.141834 30318 raft_consensus.cc:504] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.141973 30318 leader_election.cc:288] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Requested vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:27.142119 30247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587aded043a4d3fa707c0528bbedfcf" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da"
I20260928 02:43:27.142184 30247 raft_consensus.cc:3037] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.142131 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2587aded043a4d3fa707c0528bbedfcf" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
I20260928 02:43:27.142216 29967 raft_consensus.cc:3037] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.142867 29967 raft_consensus.cc:2454] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:27.142967 30247 raft_consensus.cc:2454] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:27.143085 30039 leader_election.cc:302] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:27.143208 30318 raft_consensus.cc:2781] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:27.143419 30318 raft_consensus.cc:686] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Running, Role: LEADER
I20260928 02:43:27.143529 30318 consensus_queue.cc:237] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.143973 30335 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: TxnStatusTablet state changed. Reason: New leader 7d9fcf5b339c4c46b33e9b40a51bfaf6. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.144140 30335 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:27.144367 30337 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:43:27.144476 30337 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:43:27.144337 29810 catalog_manager.cc:5495] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130). New cstate: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:27.162184 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.159694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.0.0.1:57680)
0928 02:43:27.159770 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.159782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.159831 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.160002 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.160002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.160003 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:27.160063 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.160065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.160690 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.160691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.161258 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.161259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.161637 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.161638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.161717 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.161735 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.162000 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.162010 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.162067 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.162112 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:43:27.162184 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.159739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.27.109.130:45937)
0928 02:43:27.159779 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.159782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.159812 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:43:27.159922 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.159923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.159938 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.159989 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.160140 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.160607 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.161357 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.161560 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.161634 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.161835 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.161933 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.162021 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.162071 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.162091 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:43:27.163782 30247 raft_consensus.cc:1261] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.163822 29967 raft_consensus.cc:1261] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.164121 30335 consensus_queue.cc:1035] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:27.164261 30318 consensus_queue.cc:1035] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:27.165374 30320 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: TxnStatusTablet state changed. Reason: New leader 7d9fcf5b339c4c46b33e9b40a51bfaf6. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.165464 30320 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:27.166456 30321 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: TxnStatusTablet state changed. Reason: New leader 7d9fcf5b339c4c46b33e9b40a51bfaf6. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.166558 30321 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:27.167107 30335 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.167200 30335 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:27.167243 30335 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.167280 30335 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:43:27.167380 30320 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.167465 30320 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P d066ce42e89a43ec9c4df0b510de3ae8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:27.167768 30321 tablet_replica.cc:439] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } } }
I20260928 02:43:27.167852 30321 tablet_replica.cc:442] T 2587aded043a4d3fa707c0528bbedfcf P c09185879593487389a76d0571aab0da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:43:27.168002 30341 mvcc.cc:205] Tried to move back new op lower bound from 7334147715740389376 to 7334147715660423168. Current Snapshot: MvccSnapshot[applied={T|T < 7334147715740389376}]
I20260928 02:43:27.168133 30343 mvcc.cc:205] Tried to move back new op lower bound from 7334147715740389376 to 7334147715660423168. Current Snapshot: MvccSnapshot[applied={T|T < 7334147715740389376}]
I20260928 02:43:27.172400 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.169956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.27.109.131:34081)
0928 02:43:27.169996 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.169998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.170015 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:43:27.170044 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.170045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.170061 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.170154 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.170263 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.170807 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.171685 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.171870 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.171943 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.172098 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.172178 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.172261 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.172303 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.172321 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:27.176805 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.174679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41483 (local address 127.27.109.130:45937)
0928 02:43:27.174717 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.174719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.174727 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:27.174816 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.174817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.174827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.174880 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.174986 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.175413 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.176119 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.176324 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.176405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.176578 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.176607 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.176677 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.176727 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.176745 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:27.176824 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.174573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.27.109.131:41483)
0928 02:43:27.174651 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.174664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.174764 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.174883 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.174884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.174885 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:27.174934 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.174935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.175491 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.175492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.176037 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.176038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.176407 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.176408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.176470 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.176483 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.176677 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.176679 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.176726 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.176762 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:43:27.179104 29967 raft_consensus.cc:1261] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Refusing update from remote peer c09185879593487389a76d0571aab0da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.179188 30107 raft_consensus.cc:1261] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Refusing update from remote peer c09185879593487389a76d0571aab0da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.179354 30321 consensus_queue.cc:1035] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [LEADER]: Connected to new peer: Peer: permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:27.179466 30323 consensus_queue.cc:1035] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:27.181210 30340 mvcc.cc:205] Tried to move back new op lower bound from 7334147715803885568 to 7334147714828349440. Current Snapshot: MvccSnapshot[applied={T|T < 7334147715803885568}]
I20260928 02:43:27.187397 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.184808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:34081 (local address 127.27.109.130:58653)
0928 02:43:27.184924 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.184934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.184964 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.185109 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.185110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.185111 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:27.185166 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.185167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.185976 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.185977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.186603 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.186605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.187044 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.187045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.187105 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.187130 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.187295 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.187296 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.187326 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.187352 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:43:27.187419 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.184907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:58653 (local address 127.27.109.131:34081)
0928 02:43:27.184943 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.184945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.184971 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:43:27.185028 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.185029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.185038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.185101 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.185206 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.185895 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.186687 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.186950 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.187012 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.187203 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.187237 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.187296 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.187328 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.187347 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:27.194021 29810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:27.194120 29810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:27.194828 29810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Sending DeleteTablet for 3 replicas of tablet 11f83539d78e46cc98ab1ecb8d2c5432
I20260928 02:43:27.195092 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:27 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:36782
I20260928 02:43:27.195161 30087 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:27 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:57666
I20260928 02:43:27.195319 30358 tablet_replica.cc:330] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: stopping tablet replica
I20260928 02:43:27.195433 30358 raft_consensus.cc:2229] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:27.195525 30358 raft_consensus.cc:2258] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:27.195616 30227 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f83539d78e46cc98ab1ecb8d2c5432 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:27 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45874
I20260928 02:43:27.195844 30360 tablet_replica.cc:330] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: stopping tablet replica
I20260928 02:43:27.195955 30360 raft_consensus.cc:2229] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:27.196074 30360 raft_consensus.cc:2258] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:27.196133 29810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:27.196270 29810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:27.196956 30358 ts_tablet_manager.cc:1905] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:27.197481 29947 tablet_service.cc:1467] Processing CreateTablet for tablet d2b2f04361624541b372367e60689377 (DEFAULT_TABLE table=test-table [id=2587faccf389492a8598718a00fe32c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:27.197486 30227 tablet_service.cc:1467] Processing CreateTablet for tablet d2b2f04361624541b372367e60689377 (DEFAULT_TABLE table=test-table [id=2587faccf389492a8598718a00fe32c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:27.197618 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b2f04361624541b372367e60689377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:27.197634 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b2f04361624541b372367e60689377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:27.198524 30314 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Bootstrap starting.
I20260928 02:43:27.198854 30317 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Bootstrap starting.
I20260928 02:43:27.198946 30314 tablet_bootstrap.cc:654] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:27.199241 30317 tablet_bootstrap.cc:654] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:27.199776 30087 tablet_service.cc:1467] Processing CreateTablet for tablet d2b2f04361624541b372367e60689377 (DEFAULT_TABLE table=test-table [id=2587faccf389492a8598718a00fe32c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:27.199888 30317 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: No bootstrap required, opened a new log
I20260928 02:43:27.199885 30087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b2f04361624541b372367e60689377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:27.199927 30317 ts_tablet_manager.cc:1397] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:27.200067 30317 raft_consensus.cc:348] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.200124 30317 raft_consensus.cc:374] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:27.200142 30317 raft_consensus.cc:729] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09185879593487389a76d0571aab0da, State: Initialized, Role: FOLLOWER
I20260928 02:43:27.200209 30317 consensus_queue.cc:260] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.200289 30317 ts_tablet_manager.cc:1428] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:27.201042 30360 ts_tablet_manager.cc:1905] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:27.201164 30313 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Bootstrap starting.
I20260928 02:43:27.201534 30313 tablet_bootstrap.cc:654] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:27.201894 30358 ts_tablet_manager.cc:1918] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:27.202100 30313 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: No bootstrap required, opened a new log
I20260928 02:43:27.202168 30313 ts_tablet_manager.cc:1397] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:27.202302 30313 raft_consensus.cc:348] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.202345 30313 raft_consensus.cc:374] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:27.202358 30313 raft_consensus.cc:729] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Initialized, Role: FOLLOWER
I20260928 02:43:27.202412 30313 consensus_queue.cc:260] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.202527 30313 ts_tablet_manager.cc:1428] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:27.202544 30359 tablet_replica.cc:330] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: stopping tablet replica
I20260928 02:43:27.202656 30359 raft_consensus.cc:2229] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:27.202728 30359 raft_consensus.cc:2258] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:27.202791 30360 ts_tablet_manager.cc:1918] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:27.202912 30360 log.cc:1196] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/11f83539d78e46cc98ab1ecb8d2c5432
I20260928 02:43:27.202870 30314 tablet_bootstrap.cc:492] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: No bootstrap required, opened a new log
I20260928 02:43:27.203184 30360 ts_tablet_manager.cc:1939] T 11f83539d78e46cc98ab1ecb8d2c5432 P c09185879593487389a76d0571aab0da: Deleting consensus metadata
I20260928 02:43:27.203181 30314 ts_tablet_manager.cc:1397] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:43:27.203325 30314 raft_consensus.cc:348] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.203369 30314 raft_consensus.cc:374] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:27.203382 30314 raft_consensus.cc:729] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Initialized, Role: FOLLOWER
I20260928 02:43:27.203447 30314 consensus_queue.cc:260] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.203510 29794 catalog_manager.cc:4841] TS c09185879593487389a76d0571aab0da (127.27.109.131:34081): tablet 11f83539d78e46cc98ab1ecb8d2c5432 (table test-table [id=8eb0a0b123734b24816d50227cfee915]) successfully deleted
I20260928 02:43:27.203540 30314 ts_tablet_manager.cc:1428] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:27.203670 30358 log.cc:1196] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/11f83539d78e46cc98ab1ecb8d2c5432
I20260928 02:43:27.203982 30358 ts_tablet_manager.cc:1939] T 11f83539d78e46cc98ab1ecb8d2c5432 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting consensus metadata
I20260928 02:43:27.204445 29795 catalog_manager.cc:4841] TS d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881): tablet 11f83539d78e46cc98ab1ecb8d2c5432 (table test-table [id=8eb0a0b123734b24816d50227cfee915]) successfully deleted
I20260928 02:43:27.206806 30359 ts_tablet_manager.cc:1905] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:27.208788 30359 ts_tablet_manager.cc:1918] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:27.208861 30359 log.cc:1196] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/11f83539d78e46cc98ab1ecb8d2c5432
I20260928 02:43:27.209103 30359 ts_tablet_manager.cc:1939] T 11f83539d78e46cc98ab1ecb8d2c5432 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting consensus metadata
I20260928 02:43:27.209502 29795 catalog_manager.cc:4841] TS 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937): tablet 11f83539d78e46cc98ab1ecb8d2c5432 (table test-table [id=8eb0a0b123734b24816d50227cfee915]) successfully deleted
I20260928 02:43:27.227665 30318 raft_consensus.cc:482] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:27.227748 30318 raft_consensus.cc:504] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.227880 30318 leader_election.cc:288] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:27.228070 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b2f04361624541b372367e60689377" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" is_pre_election: true
I20260928 02:43:27.228148 29967 raft_consensus.cc:2454] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:27.228094 30247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b2f04361624541b372367e60689377" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da" is_pre_election: true
I20260928 02:43:27.228194 30247 raft_consensus.cc:2454] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:27.228274 30039 leader_election.cc:302] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:27.228344 30318 raft_consensus.cc:2781] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:27.228372 30318 raft_consensus.cc:482] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:27.228391 30318 raft_consensus.cc:3037] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.229171 30318 raft_consensus.cc:504] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.229327 30318 leader_election.cc:288] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Requested vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:27.229472 30247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b2f04361624541b372367e60689377" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da"
I20260928 02:43:27.229537 30247 raft_consensus.cc:3037] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.229507 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b2f04361624541b372367e60689377" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
I20260928 02:43:27.229568 29967 raft_consensus.cc:3037] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:27.230273 29967 raft_consensus.cc:2454] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:27.230289 30247 raft_consensus.cc:2454] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:27.230409 30039 leader_election.cc:302] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:27.230511 30318 raft_consensus.cc:2781] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:27.230552 30318 raft_consensus.cc:686] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Running, Role: LEADER
I20260928 02:43:27.230594 30318 consensus_queue.cc:237] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:27.231061 29810 catalog_manager.cc:5495] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130). New cstate: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:27.241995 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.239120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.27.109.130:45937)
0928 02:43:27.239157 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.239159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.239170 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:27.239257 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.239258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.239268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.239328 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.239449 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.240192 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.241220 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.241520 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.241591 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.241753 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.241785 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.241838 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.241908 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.241925 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:27.247042 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.244095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.27.109.130:47209)
0928 02:43:27.244180 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:27.244189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:27.244224 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:27.244454 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:27.244454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:27.244455 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:27.244486 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.244487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.245371 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.245372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.246208 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:27.246210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.246704 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.246705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:27.246752 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.246755 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.246895 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.246895 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 02:43:27.246930 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 02:43:27.246965 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:43:27.247360 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:27.244314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:47209 (local address 127.27.109.130:45937)
0928 02:43:27.244354 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:27.244355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:27.244361 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:43:27.244377 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:27.244378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:27.244385 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:27.244426 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:27.244541 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.245251 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.246291 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:27.246608 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:27.246678 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:27.246814 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:27.246841 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:27.246878 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:27.247288 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 02:43:27.247303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:43:27.249378 29967 raft_consensus.cc:1261] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.249405 30247 raft_consensus.cc:1261] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:27.249565 30318 consensus_queue.cc:1035] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:27.249742 30318 consensus_queue.cc:1035] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:43:27.254642 30340 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:43:28.775127 29810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:28.775226 29810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:28.776105 29810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Sending DeleteTablet for 3 replicas of tablet d2b2f04361624541b372367e60689377
I20260928 02:43:28.776360 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b2f04361624541b372367e60689377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:28 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:36782
I20260928 02:43:28.776383 30086 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b2f04361624541b372367e60689377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:28 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:57666
I20260928 02:43:28.776420 30227 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b2f04361624541b372367e60689377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:28 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45874
I20260928 02:43:28.776692 30387 tablet_replica.cc:330] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: stopping tablet replica
I20260928 02:43:28.776770 30387 raft_consensus.cc:2229] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:28.776824 30387 raft_consensus.cc:2258] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:28.776933 30388 tablet_replica.cc:330] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: stopping tablet replica
I20260928 02:43:28.776996 30388 raft_consensus.cc:2229] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:28.777081 30388 raft_consensus.cc:2258] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:28.777285 30387 ts_tablet_manager.cc:1905] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:28.777300 30388 ts_tablet_manager.cc:1905] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:28.779070 30388 ts_tablet_manager.cc:1918] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:43:28.779129 30388 log.cc:1196] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/d2b2f04361624541b372367e60689377
I20260928 02:43:28.779278 30387 ts_tablet_manager.cc:1918] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:43:28.779342 30387 log.cc:1196] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/d2b2f04361624541b372367e60689377
I20260928 02:43:28.779598 30387 ts_tablet_manager.cc:1939] T d2b2f04361624541b372367e60689377 P d066ce42e89a43ec9c4df0b510de3ae8: Deleting consensus metadata
I20260928 02:43:28.779866 30389 tablet_replica.cc:330] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: stopping tablet replica
I20260928 02:43:28.779925 29795 catalog_manager.cc:4841] TS d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881): tablet d2b2f04361624541b372367e60689377 (table test-table [id=2587faccf389492a8598718a00fe32c8]) successfully deleted
I20260928 02:43:28.779953 30389 raft_consensus.cc:2229] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:28.779992 30389 raft_consensus.cc:2258] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:28.780265 30389 ts_tablet_manager.cc:1905] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:28.781641 30388 ts_tablet_manager.cc:1939] T d2b2f04361624541b372367e60689377 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting consensus metadata
I20260928 02:43:28.781904 30389 ts_tablet_manager.cc:1918] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:43:28.781971 29795 catalog_manager.cc:4841] TS 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937): tablet d2b2f04361624541b372367e60689377 (table test-table [id=2587faccf389492a8598718a00fe32c8]) successfully deleted
I20260928 02:43:28.782006 30389 log.cc:1196] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/d2b2f04361624541b372367e60689377
I20260928 02:43:28.782217 30389 ts_tablet_manager.cc:1939] T d2b2f04361624541b372367e60689377 P c09185879593487389a76d0571aab0da: Deleting consensus metadata
I20260928 02:43:28.782646 29794 catalog_manager.cc:4841] TS c09185879593487389a76d0571aab0da (127.27.109.131:34081): tablet d2b2f04361624541b372367e60689377 (table test-table [id=2587faccf389492a8598718a00fe32c8]) successfully deleted
Sep 28 02:43:28 dist-test-slave-b5p5 krb5kdc[29765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563408, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:43:28 dist-test-slave-b5p5 krb5kdc[29765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563408, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:28.800824 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:28.793073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.27.109.190:33657)
0928 02:43:28.793260 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:28.793262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:28.793272 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:28.793296 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:28.793296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:28.793307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:28.793384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:28.793951 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.794640 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.795416 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.795666 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.795751 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:28.798112 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:28.798114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:28.798114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:28.798121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:28.799630 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:28.799950 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:28.799951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:28.799952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:28.800017 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:28.800203 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:28.800203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:28.800203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:28.800315 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:28.800398 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:28.800557 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:43:28.800682 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260928 02:43:28.802549 29810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:28.802711 29810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:28.804178 29947 tablet_service.cc:1467] Processing CreateTablet for tablet aeb2c158d9e74acfba729e27da7de9bf (DEFAULT_TABLE table=test-table [id=a84d447004a94707997d6746eeffec8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:28.804204 30086 tablet_service.cc:1467] Processing CreateTablet for tablet aeb2c158d9e74acfba729e27da7de9bf (DEFAULT_TABLE table=test-table [id=a84d447004a94707997d6746eeffec8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:28.804226 30227 tablet_service.cc:1467] Processing CreateTablet for tablet aeb2c158d9e74acfba729e27da7de9bf (DEFAULT_TABLE table=test-table [id=a84d447004a94707997d6746eeffec8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:28.804312 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb2c158d9e74acfba729e27da7de9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:28.804312 30086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb2c158d9e74acfba729e27da7de9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:28.804364 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb2c158d9e74acfba729e27da7de9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:28.805809 30399 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Bootstrap starting.
I20260928 02:43:28.805925 30398 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Bootstrap starting.
I20260928 02:43:28.805989 30400 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Bootstrap starting.
I20260928 02:43:28.806211 30399 tablet_bootstrap.cc:654] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:28.806349 30400 tablet_bootstrap.cc:654] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:28.806349 30398 tablet_bootstrap.cc:654] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:28.806898 30399 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: No bootstrap required, opened a new log
I20260928 02:43:28.806959 30399 ts_tablet_manager.cc:1397] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:28.806970 30398 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: No bootstrap required, opened a new log
I20260928 02:43:28.807010 30398 ts_tablet_manager.cc:1397] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:28.807107 30399 raft_consensus.cc:348] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807130 30398 raft_consensus.cc:348] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807180 30399 raft_consensus.cc:374] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:28.807188 30398 raft_consensus.cc:374] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:28.807205 30399 raft_consensus.cc:729] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Initialized, Role: FOLLOWER
I20260928 02:43:28.807194 30400 tablet_bootstrap.cc:492] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: No bootstrap required, opened a new log
I20260928 02:43:28.807211 30398 raft_consensus.cc:729] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09185879593487389a76d0571aab0da, State: Initialized, Role: FOLLOWER
I20260928 02:43:28.807248 30400 ts_tablet_manager.cc:1397] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:28.807253 30399 consensus_queue.cc:260] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807272 30398 consensus_queue.cc:260] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807376 30400 raft_consensus.cc:348] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807420 30398 ts_tablet_manager.cc:1428] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:28.807420 30399 ts_tablet_manager.cc:1428] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:28.807449 30400 raft_consensus.cc:374] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:28.807473 30400 raft_consensus.cc:729] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Initialized, Role: FOLLOWER
I20260928 02:43:28.807533 30400 consensus_queue.cc:260] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.807644 30400 ts_tablet_manager.cc:1428] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:28.994248 30404 raft_consensus.cc:482] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:28.994344 30404 raft_consensus.cc:504] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.994755 30404 leader_election.cc:288] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:28.998155 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:28.994946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.27.109.129:33939)
0928 02:43:28.995113 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:28.995123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:28.995208 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:28.995410 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:28.995410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:28.995412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:28.995475 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:28.995477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.996306 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.996307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:28.997234 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:28.997236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.997888 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.997889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:28.997958 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:28.997962 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:28.998023 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:43:28.998062 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260928 02:43:28.998149 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:28.995117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33939 (local address 127.27.109.130:45937)
0928 02:43:28.995255 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:28.995258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:28.995267 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:28.995298 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:28.995299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:28.995310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:28.995406 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:28.995542 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.996248 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.997328 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.997816 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.997871 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:28.997878 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:28.998056 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:43:28.998093 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 02:43:28.998541 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:28.994948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:34081 (local address 127.27.109.129:33547)
0928 02:43:28.995280 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:28.995287 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:28.995320 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:28.995491 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:28.995493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:28.995495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:28.995890 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:28.995893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.996724 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.996725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:28.997563 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:28.997565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.998228 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.998229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:28.998315 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:28.998319 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:28.998367 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:43:28.998400 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260928 02:43:28.998512 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb2c158d9e74acfba729e27da7de9bf" candidate_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" is_pre_election: true
I20260928 02:43:28.998610 30107 raft_consensus.cc:2454] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d066ce42e89a43ec9c4df0b510de3ae8 in term 0.
I20260928 02:43:28.998699 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:28.995145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33547 (local address 127.27.109.131:34081)
0928 02:43:28.995277 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:28.995279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:28.995319 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:43:28.995361 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:28.995362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:28.995372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:28.995435 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:28.995982 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.996639 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.997656 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:28.998139 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:28.998527 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:28.998537 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:28.998561 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 02:43:28.998616 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 02:43:28.998795 29899 leader_election.cc:302] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:28.998932 30404 raft_consensus.cc:2781] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:28.998979 30404 raft_consensus.cc:482] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:28.999006 30404 raft_consensus.cc:3037] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:28.999061 30246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb2c158d9e74acfba729e27da7de9bf" candidate_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da" is_pre_election: true
I20260928 02:43:28.999181 30246 raft_consensus.cc:2454] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d066ce42e89a43ec9c4df0b510de3ae8 in term 0.
I20260928 02:43:28.999742 30404 raft_consensus.cc:504] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:28.999876 30404 leader_election.cc:288] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [CANDIDATE]: Term 1 election: Requested vote from peers 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:28.999998 30246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb2c158d9e74acfba729e27da7de9bf" candidate_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da"
I20260928 02:43:28.999996 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb2c158d9e74acfba729e27da7de9bf" candidate_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6"
I20260928 02:43:29.000066 30107 raft_consensus.cc:3037] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:29.000066 30246 raft_consensus.cc:3037] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:29.000836 30107 raft_consensus.cc:2454] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d066ce42e89a43ec9c4df0b510de3ae8 in term 1.
I20260928 02:43:29.000836 30246 raft_consensus.cc:2454] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d066ce42e89a43ec9c4df0b510de3ae8 in term 1.
I20260928 02:43:29.001035 29898 leader_election.cc:302] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09185879593487389a76d0571aab0da, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:29.001149 30404 raft_consensus.cc:2781] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:29.001286 30404 raft_consensus.cc:686] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 LEADER]: Becoming Leader. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Running, Role: LEADER
I20260928 02:43:29.001361 30404 consensus_queue.cc:237] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.002058 29809 catalog_manager.cc:5495] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129). New cstate: current_term: 1 leader_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:29.035552 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.033251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.27.109.129:33881)
0928 02:43:29.033405 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:29.033406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:29.033413 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:29.033445 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:29.033445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:29.033462 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:29.033534 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:29.033645 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.034116 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.034838 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.035036 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.035100 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.035263 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.035373 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.035432 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:29.035474 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:43:29.035489 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260928 02:43:29.040180 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.037877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:56937 (local address 127.27.109.130:45937)
0928 02:43:29.037933 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:29.037935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:29.037944 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:29.038012 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:29.038013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:29.038023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:29.038105 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:29.038250 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.038777 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.039508 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.039703 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.039778 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.039945 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.039976 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.040034 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:29.040090 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:43:29.040110 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:43:29.040217 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.037680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:45937 (local address 127.27.109.129:56937)
0928 02:43:29.037864 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:29.037882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:29.037951 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:29.038113 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:29.038113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:29.038115 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:29.038177 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:29.038178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.038868 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.038869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:29.039426 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:29.039427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.039777 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.039778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:29.039838 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.039858 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.040037 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.040039 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:29.040085 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:43:29.040125 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260928 02:43:29.042140 30107 raft_consensus.cc:1261] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Refusing update from remote peer d066ce42e89a43ec9c4df0b510de3ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:29.042152 30246 raft_consensus.cc:1261] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Refusing update from remote peer d066ce42e89a43ec9c4df0b510de3ae8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:29.042436 30410 consensus_queue.cc:1035] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:29.042585 30404 consensus_queue.cc:1035] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:29.049137 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.046937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:36555 (local address 127.27.109.129:33881)
0928 02:43:29.046976 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:29.046978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:29.047099 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:43:29.047124 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:29.047124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:29.047134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:29.047184 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:29.047272 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.047719 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.048456 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.048706 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.048763 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.048921 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.048951 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.049003 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:29.049069 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:43:29.049093 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:29.049181 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.046880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:33881 (local address 127.27.109.130:36555)
0928 02:43:29.047010 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:29.047020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:29.047077 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:29.047187 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:29.047187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:29.047188 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:43:29.047226 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:29.047227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.047803 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.047804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:29.048356 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:29.048358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.048775 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.048776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:29.048829 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.048833 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.049007 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.049009 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:29.049066 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:43:29.049113 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260928 02:43:29.055393 29809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:29.055509 29809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:29.056118 29809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Sending DeleteTablet for 3 replicas of tablet aeb2c158d9e74acfba729e27da7de9bf
I20260928 02:43:29.056432 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb2c158d9e74acfba729e27da7de9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:29 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:36782
I20260928 02:43:29.056428 30227 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb2c158d9e74acfba729e27da7de9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:29 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45874
I20260928 02:43:29.056519 30389 tablet_replica.cc:330] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: stopping tablet replica
I20260928 02:43:29.056520 30086 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb2c158d9e74acfba729e27da7de9bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:29 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:57666
I20260928 02:43:29.056576 30389 raft_consensus.cc:2229] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:29.056593 30387 tablet_replica.cc:330] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: stopping tablet replica
I20260928 02:43:29.056622 30389 raft_consensus.cc:2258] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:29.056635 30387 raft_consensus.cc:2229] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:29.056710 30387 raft_consensus.cc:2258] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:29.056790 30388 tablet_replica.cc:330] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: stopping tablet replica
I20260928 02:43:29.056843 30388 raft_consensus.cc:2229] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:29.056875 30388 raft_consensus.cc:2258] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:29.056946 30389 ts_tablet_manager.cc:1905] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:29.056898 29809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:29.057034 29809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:29.057161 30387 ts_tablet_manager.cc:1905] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:29.057654 30388 ts_tablet_manager.cc:1905] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:29.058558 30389 ts_tablet_manager.cc:1918] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:29.058609 30389 log.cc:1196] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/aeb2c158d9e74acfba729e27da7de9bf
I20260928 02:43:29.058667 30387 ts_tablet_manager.cc:1918] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:29.058735 30387 log.cc:1196] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/aeb2c158d9e74acfba729e27da7de9bf
I20260928 02:43:29.058950 30387 ts_tablet_manager.cc:1939] T aeb2c158d9e74acfba729e27da7de9bf P d066ce42e89a43ec9c4df0b510de3ae8: Deleting consensus metadata
I20260928 02:43:29.059005 30389 ts_tablet_manager.cc:1939] T aeb2c158d9e74acfba729e27da7de9bf P c09185879593487389a76d0571aab0da: Deleting consensus metadata
I20260928 02:43:29.059267 30388 ts_tablet_manager.cc:1918] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:43:29.059315 30388 log.cc:1196] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/aeb2c158d9e74acfba729e27da7de9bf
I20260928 02:43:29.059373 29794 catalog_manager.cc:4841] TS c09185879593487389a76d0571aab0da (127.27.109.131:34081): tablet aeb2c158d9e74acfba729e27da7de9bf (table test-table [id=a84d447004a94707997d6746eeffec8b]) successfully deleted
I20260928 02:43:29.059516 30388 ts_tablet_manager.cc:1939] T aeb2c158d9e74acfba729e27da7de9bf P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting consensus metadata
I20260928 02:43:29.059875 29795 catalog_manager.cc:4841] TS 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937): tablet aeb2c158d9e74acfba729e27da7de9bf (table test-table [id=a84d447004a94707997d6746eeffec8b]) successfully deleted
I20260928 02:43:29.059947 29795 catalog_manager.cc:4841] TS d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881): tablet aeb2c158d9e74acfba729e27da7de9bf (table test-table [id=a84d447004a94707997d6746eeffec8b]) successfully deleted
I20260928 02:43:29.060423 30086 tablet_service.cc:1467] Processing CreateTablet for tablet b9140d3977354471a143604bf336c375 (DEFAULT_TABLE table=test-table [id=fa56ecf077d4481981469ea28b09d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:29.060429 30227 tablet_service.cc:1467] Processing CreateTablet for tablet b9140d3977354471a143604bf336c375 (DEFAULT_TABLE table=test-table [id=fa56ecf077d4481981469ea28b09d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:29.060539 30086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9140d3977354471a143604bf336c375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:29.060557 30227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9140d3977354471a143604bf336c375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:29.061334 30398 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Bootstrap starting.
I20260928 02:43:29.061450 29947 tablet_service.cc:1467] Processing CreateTablet for tablet b9140d3977354471a143604bf336c375 (DEFAULT_TABLE table=test-table [id=fa56ecf077d4481981469ea28b09d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:29.061553 29947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9140d3977354471a143604bf336c375. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:29.061642 30398 tablet_bootstrap.cc:654] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:29.061743 30400 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Bootstrap starting.
I20260928 02:43:29.062031 30400 tablet_bootstrap.cc:654] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:29.062196 30398 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: No bootstrap required, opened a new log
I20260928 02:43:29.062237 30398 ts_tablet_manager.cc:1397] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:29.062347 30398 raft_consensus.cc:348] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.062422 30398 raft_consensus.cc:374] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:29.062443 30398 raft_consensus.cc:729] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09185879593487389a76d0571aab0da, State: Initialized, Role: FOLLOWER
I20260928 02:43:29.062506 30398 consensus_queue.cc:260] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.062602 30398 ts_tablet_manager.cc:1428] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:29.062847 30399 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8: Bootstrap starting.
I20260928 02:43:29.063254 30399 tablet_bootstrap.cc:654] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:29.063748 30400 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: No bootstrap required, opened a new log
I20260928 02:43:29.063794 30400 ts_tablet_manager.cc:1397] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:43:29.063865 30400 raft_consensus.cc:348] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.063939 30400 raft_consensus.cc:374] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:29.063958 30400 raft_consensus.cc:729] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Initialized, Role: FOLLOWER
I20260928 02:43:29.063997 30400 consensus_queue.cc:260] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.064077 30400 ts_tablet_manager.cc:1428] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:29.064307 30399 tablet_bootstrap.cc:492] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8: No bootstrap required, opened a new log
I20260928 02:43:29.064348 30399 ts_tablet_manager.cc:1397] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:43:29.064507 30399 raft_consensus.cc:348] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.064580 30399 raft_consensus.cc:374] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:29.064596 30399 raft_consensus.cc:729] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d066ce42e89a43ec9c4df0b510de3ae8, State: Initialized, Role: FOLLOWER
I20260928 02:43:29.064649 30399 consensus_queue.cc:260] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.064749 30399 ts_tablet_manager.cc:1428] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:29.079228 30370 raft_consensus.cc:482] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:29.079301 30370 raft_consensus.cc:504] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.079439 30370 leader_election.cc:288] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:29.079599 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9140d3977354471a143604bf336c375" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" is_pre_election: true
I20260928 02:43:29.079680 29967 raft_consensus.cc:2454] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:29.079773 30246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9140d3977354471a143604bf336c375" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da" is_pre_election: true
I20260928 02:43:29.079852 30246 raft_consensus.cc:2454] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 0.
I20260928 02:43:29.079816 30039 leader_election.cc:302] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, d066ce42e89a43ec9c4df0b510de3ae8; no voters:
I20260928 02:43:29.079936 30370 raft_consensus.cc:2781] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:29.079977 30370 raft_consensus.cc:482] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:29.080004 30370 raft_consensus.cc:3037] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:29.080605 30370 raft_consensus.cc:504] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.080775 30370 leader_election.cc:288] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Requested vote from peers d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881), c09185879593487389a76d0571aab0da (127.27.109.131:34081)
I20260928 02:43:29.080899 30246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9140d3977354471a143604bf336c375" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c09185879593487389a76d0571aab0da"
I20260928 02:43:29.080974 30246 raft_consensus.cc:3037] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:29.080981 29967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9140d3977354471a143604bf336c375" candidate_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d066ce42e89a43ec9c4df0b510de3ae8"
I20260928 02:43:29.081058 29967 raft_consensus.cc:3037] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:29.081547 30246 raft_consensus.cc:2454] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:29.081712 30038 leader_election.cc:302] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9fcf5b339c4c46b33e9b40a51bfaf6, c09185879593487389a76d0571aab0da; no voters:
I20260928 02:43:29.081792 29967 raft_consensus.cc:2454] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9fcf5b339c4c46b33e9b40a51bfaf6 in term 1.
I20260928 02:43:29.081812 30370 raft_consensus.cc:2781] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:29.081852 30370 raft_consensus.cc:686] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9fcf5b339c4c46b33e9b40a51bfaf6, State: Running, Role: LEADER
I20260928 02:43:29.081904 30370 consensus_queue.cc:237] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } }
I20260928 02:43:29.082500 29809 catalog_manager.cc:5495] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130). New cstate: current_term: 1 leader_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d9fcf5b339c4c46b33e9b40a51bfaf6" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 45937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:29.088806 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:29.086277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.27.109.130:45937)
0928 02:43:29.086317 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:29.086319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:29.086352 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:29.086426 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:29.086427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:29.086437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:29.086561 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:29.086676 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.087122 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.088154 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:29.088327 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:29.088397 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:29.088543 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:29.088596 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:29.088634 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:29.088690 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:43:29.088708 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:29.090852 30246 raft_consensus.cc:1261] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:29.091019 29967 raft_consensus.cc:1261] T b9140d3977354471a143604bf336c375 P d066ce42e89a43ec9c4df0b510de3ae8 [term 1 FOLLOWER]: Refusing update from remote peer 7d9fcf5b339c4c46b33e9b40a51bfaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:29.091048 30375 consensus_queue.cc:1035] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c09185879593487389a76d0571aab0da" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:29.091198 30375 consensus_queue.cc:1035] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d066ce42e89a43ec9c4df0b510de3ae8" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:43:29.095137 30368 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:43:30.611392 29809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:30.611485 29809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:30.612190 29809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c759d6a1aca46c6b8e64d25d012bcbd: Sending DeleteTablet for 3 replicas of tablet b9140d3977354471a143604bf336c375
I20260928 02:43:30.612429 30227 tablet_service.cc:1514] Processing DeleteTablet for tablet b9140d3977354471a143604bf336c375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:30 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45874
I20260928 02:43:30.612445 30086 tablet_service.cc:1514] Processing DeleteTablet for tablet b9140d3977354471a143604bf336c375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:30 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:57666
I20260928 02:43:30.612471 29947 tablet_service.cc:1514] Processing DeleteTablet for tablet b9140d3977354471a143604bf336c375 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:30 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:36782
I20260928 02:43:30.612542 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29882
I20260928 02:43:30.613332 30449 tablet_replica.cc:330] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: stopping tablet replica
I20260928 02:43:30.613407 30449 raft_consensus.cc:2229] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:30.613446 30449 raft_consensus.cc:2258] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:30.613606 30450 tablet_replica.cc:330] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: stopping tablet replica
I20260928 02:43:30.613669 30450 raft_consensus.cc:2229] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:30.613695 30449 ts_tablet_manager.cc:1905] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:30.613813 30450 raft_consensus.cc:2258] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:30.614073 30450 ts_tablet_manager.cc:1905] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:30.615583 30449 ts_tablet_manager.cc:1918] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:43:30.615674 30449 log.cc:1196] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/b9140d3977354471a143604bf336c375
I20260928 02:43:30.615931 30449 ts_tablet_manager.cc:1939] T b9140d3977354471a143604bf336c375 P c09185879593487389a76d0571aab0da: Deleting consensus metadata
I20260928 02:43:30.616026 30450 ts_tablet_manager.cc:1918] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:43:30.616101 30450 log.cc:1196] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/b9140d3977354471a143604bf336c375
I20260928 02:43:30.616250 29794 catalog_manager.cc:4841] TS c09185879593487389a76d0571aab0da (127.27.109.131:34081): tablet b9140d3977354471a143604bf336c375 (table test-table [id=fa56ecf077d4481981469ea28b09d389]) successfully deleted
I20260928 02:43:30.616482 30450 ts_tablet_manager.cc:1939] T b9140d3977354471a143604bf336c375 P 7d9fcf5b339c4c46b33e9b40a51bfaf6: Deleting consensus metadata
I20260928 02:43:30.616791 29795 catalog_manager.cc:4841] TS 7d9fcf5b339c4c46b33e9b40a51bfaf6 (127.27.109.130:45937): tablet b9140d3977354471a143604bf336c375 (table test-table [id=fa56ecf077d4481981469ea28b09d389]) successfully deleted
W20260928 02:43:30.620455 29795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:33881 (error 108)
I20260928 02:43:30.620690 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30022
I20260928 02:43:30.621688 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:30.620808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:33881 (local address 127.0.0.1:36814)
0928 02:43:30.621580 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:30.621640 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:33881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":58,"threads_started":1}
W20260928 02:43:30.621832 29795 catalog_manager.cc:4584] TS d066ce42e89a43ec9c4df0b510de3ae8 (127.27.109.129:33881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9140d3977354471a143604bf336c375: Network error: Client connection negotiation failed: client connection to 127.27.109.129:33881: connect: Connection refused (error 111)
I20260928 02:43:30.626780 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30162
I20260928 02:43:30.632458 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 29778
2026-09-28T02:43:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5559 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:30 dist-test-slave-b5p5 krb5kdc[30456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:30 dist-test-slave-b5p5 krb5kdc[30456](info): set up 2 sockets
Sep 28 02:43:30 dist-test-slave-b5p5 krb5kdc[30456](info): commencing operation
krb5kdc: starting...
W20260928 02:43:31.670571 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:31 dist-test-slave-b5p5 krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:31.781642 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35387
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:31.860234 30469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:31.860412 30469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:31.860435 30469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:31.861475 30469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:31.861509 30469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:31.861523 30469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:31.861536 30469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:31.861547 30469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:31.862929 30469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:35387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:31.863093 30469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:31.863243 30469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:31.867838 30477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:31.868008 30475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:31.867930 30469 server_base.cc:1034] running on GCE node
W20260928 02:43:31.867823 30474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:31.868316 30469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:31.868539 30469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:31.869673 30469 hybrid_clock.cc:648] HybridClock initialized: now 1790563411869639 us; error 49 us; skew 500 ppm
Sep 28 02:43:31 dist-test-slave-b5p5 krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563411, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:31.872022 30469 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:31.872366 30469 webserver.cc:460] Webserver started at http://127.27.109.190:32865/ using document root <none> and password file <none>
I20260928 02:43:31.872581 30469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:31.872625 30469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:31.872710 30469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:31.873622 30469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "e3ad48a6d6f74a31bef16ece67700b01"
format_stamp: "Formatted at 2026-09-28 02:43:31 on dist-test-slave-b5p5"
server_key: "33ea3c8bea788f49a61fe4738f13bd9b"
server_key_iv: "46342a547c7143298cb8a3028eeb8145"
server_key_version: "encryptionkey@0"
I20260928 02:43:31.873898 30469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "e3ad48a6d6f74a31bef16ece67700b01"
format_stamp: "Formatted at 2026-09-28 02:43:31 on dist-test-slave-b5p5"
server_key: "33ea3c8bea788f49a61fe4738f13bd9b"
server_key_iv: "46342a547c7143298cb8a3028eeb8145"
server_key_version: "encryptionkey@0"
I20260928 02:43:31.875136 30469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:31.875828 30484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:31.876014 30469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:31.876093 30469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "e3ad48a6d6f74a31bef16ece67700b01"
format_stamp: "Formatted at 2026-09-28 02:43:31 on dist-test-slave-b5p5"
server_key: "33ea3c8bea788f49a61fe4738f13bd9b"
server_key_iv: "46342a547c7143298cb8a3028eeb8145"
server_key_version: "encryptionkey@0"
I20260928 02:43:31.876143 30469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:31.893759 30469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:31.895467 30469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:31.895572 30469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:31.899140 30469 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:35387
I20260928 02:43:31.899204 30536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:35387 every 8 connection(s)
I20260928 02:43:31.899529 30469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:31.900357 30537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:31.902415 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: Bootstrap starting.
I20260928 02:43:31.902981 30537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:31.903263 30537 log.cc:824] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:31.903923 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: No bootstrap required, opened a new log
I20260928 02:43:31.905422 30537 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } }
I20260928 02:43:31.905570 30537 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:31.905614 30537 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ad48a6d6f74a31bef16ece67700b01, State: Initialized, Role: FOLLOWER
I20260928 02:43:31.905777 30537 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } }
I20260928 02:43:31.905859 30537 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:31.905903 30537 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:31.905970 30537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:31.906148 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30469
I20260928 02:43:31.906224 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:31.906396 28086 external_mini_cluster.cc:1428] Setting key 19c016a1c052a5638c35ce59a53997b1
I20260928 02:43:31.906617 30537 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } }
I20260928 02:43:31.906730 30537 leader_election.cc:302] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3ad48a6d6f74a31bef16ece67700b01; no voters:
I20260928 02:43:31.906944 30537 leader_election.cc:288] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:31.907102 30543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:31.907202 30537 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:31.907259 30543 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [term 1 LEADER]: Becoming Leader. State: Replica: e3ad48a6d6f74a31bef16ece67700b01, State: Running, Role: LEADER
I20260928 02:43:31.907401 30543 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } }
I20260928 02:43:31.907758 30542 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3ad48a6d6f74a31bef16ece67700b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } } }
I20260928 02:43:31.907830 30542 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:31.907968 30544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3ad48a6d6f74a31bef16ece67700b01. Latest consensus state: current_term: 1 leader_uuid: "e3ad48a6d6f74a31bef16ece67700b01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3ad48a6d6f74a31bef16ece67700b01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35387 } } }
I20260928 02:43:31.908026 30544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01 [sys.catalog]: This master's current role is: LEADER
W20260928 02:43:31.909169 30558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:31.909224 30558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:31.909315 30547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:31.909498 30547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:31.910902 30547 catalog_manager.cc:1347] Generated new cluster ID: a94d18c581d1471c994a4cc760eb3cd5
I20260928 02:43:31.910952 30547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:31 dist-test-slave-b5p5 krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:31.918272 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:31.906829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.27.109.190:35387)
0928 02:43:31.907000 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:31.907010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:31.907045 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:43:31.907164 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:31.907171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:31.907222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:31.908233 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:31.908340 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:31.909838 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:31.910892 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:31.911487 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:31.911596 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:31.914153 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:31.914157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:31.914157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:31.914169 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:31.916730 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:31.917103 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:31.917106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:31.917107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:31.917218 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:31.917414 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:31.917414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:31.917415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:31.917706 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:31.917871 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:31.917937 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:43:31.918061 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260928 02:43:31.926151 30547 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:31.926841 30547 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:31.940081 30547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3ad48a6d6f74a31bef16ece67700b01: Generated new TSK 0
I20260928 02:43:31.940244 30547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:31.972775 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:32.051833 30563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:32.052054 30563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:32.052090 30563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:32.053220 30563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:32.053292 30563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:32.053337 30563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:32.054824 30563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:32.055068 30563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:32.055287 30563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:32.060549 30569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:32.060587 30568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:32.060606 30563 server_base.cc:1034] running on GCE node
W20260928 02:43:32.060554 30571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:32.060976 30563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:32.061170 30563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:32.062327 30563 hybrid_clock.cc:648] HybridClock initialized: now 1790563412062309 us; error 30 us; skew 500 ppm
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:32.064862 30563 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:32.065254 30563 webserver.cc:460] Webserver started at http://127.27.109.129:43789/ using document root <none> and password file <none>
I20260928 02:43:32.065474 30563 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:32.065538 30563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:32.065682 30563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:32.066784 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "b2825607b0744f87b2bad1bf2b52a527"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "a14bbdc8500cda612a3e7544838132d9"
server_key_iv: "2eb1faca5e80c6650cf96b60eefce235"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.067137 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "b2825607b0744f87b2bad1bf2b52a527"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "a14bbdc8500cda612a3e7544838132d9"
server_key_iv: "2eb1faca5e80c6650cf96b60eefce235"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.068380 30563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:32.069077 30578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.069240 30563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:32.069345 30563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "b2825607b0744f87b2bad1bf2b52a527"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "a14bbdc8500cda612a3e7544838132d9"
server_key_iv: "2eb1faca5e80c6650cf96b60eefce235"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.069417 30563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:32.093793 30563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:32.096125 30563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:32.096293 30563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:32.096525 30563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:32.096837 30563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:32.096891 30563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.096933 30563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:32.096961 30563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.102632 30563 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:42775
I20260928 02:43:32.102679 30691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:42775 every 8 connection(s)
I20260928 02:43:32.103029 30563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:32.107508 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30563
I20260928 02:43:32.107589 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:32.107800 28086 external_mini_cluster.cc:1428] Setting key 8b6197e27a26f04b00145f6ea9ab18f3
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:32.112850 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.103711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:46873 (local address 127.27.109.190:35387)
0928 02:43:32.103799 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:32.103801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:32.103960 (+ 159us) server_negotiation.cc:407] Connection header received
0928 02:43:32.104414 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:32.104415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:32.104428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:32.104484 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:32.105332 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.105814 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.106826 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.107009 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.107064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.109643 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:32.109644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:32.109645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:32.109650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:32.111438 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.111829 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.111830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.111830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.111893 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.112172 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.112173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.112174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.112321 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:32.112385 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:32.112655 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:43:32.112750 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:43:32.112982 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.103682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35387 (local address 127.27.109.129:46873)
0928 02:43:32.103834 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:32.103874 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:32.104334 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:32.104536 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:32.104539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:32.105080 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:32.105258 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.105263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.105893 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.105893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.106757 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.106759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.107081 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.107082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.107175 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.107685 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:32.107694 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:32.109560 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:32.111533 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.111538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.111540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.111752 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.111955 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.111956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.111957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.112109 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.112393 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:32.112399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:32.112520 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 02:43:32.112622 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:43:32.112768 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 02:43:32.113417 30692 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35387
I20260928 02:43:32.113509 30692 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:32.113672 30692 heartbeater.cc:511] Master 127.27.109.190:35387 requested a full tablet report, sending...
I20260928 02:43:32.114169 30501 ts_manager.cc:194] Registered new tserver with Master: b2825607b0744f87b2bad1bf2b52a527 (127.27.109.129:42775)
I20260928 02:43:32.114852 30501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:46873
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:32.132649 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:32.210765 30697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:32.210942 30697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:32.210964 30697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:32.212037 30697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:32.212072 30697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:32.212097 30697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:32.213402 30697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:32.213564 30697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:32.213722 30697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:32.218367 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:32.218499 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:32.218539 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:32.218549 30697 server_base.cc:1034] running on GCE node
I20260928 02:43:32.218832 30697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:32.219005 30697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:32.220144 30697 hybrid_clock.cc:648] HybridClock initialized: now 1790563412220130 us; error 27 us; skew 500 ppm
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:32.222609 30697 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:32.223018 30697 webserver.cc:460] Webserver started at http://127.27.109.130:41551/ using document root <none> and password file <none>
I20260928 02:43:32.223245 30697 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:32.223312 30697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:32.223449 30697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:32.224407 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "97bb5deaaae1479dae9eb61f1d32b3ed"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "199010246401eb39b668b0f07acbabae"
server_key_iv: "74fac2f2253d3b1f944bd765f2c70344"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.224771 30697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "97bb5deaaae1479dae9eb61f1d32b3ed"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "199010246401eb39b668b0f07acbabae"
server_key_iv: "74fac2f2253d3b1f944bd765f2c70344"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.225996 30697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:32.226771 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.226930 30697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:32.227025 30697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "97bb5deaaae1479dae9eb61f1d32b3ed"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "199010246401eb39b668b0f07acbabae"
server_key_iv: "74fac2f2253d3b1f944bd765f2c70344"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.227102 30697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:32.251579 30697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:32.253418 30697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:32.253576 30697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:32.253803 30697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:32.254097 30697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:32.254154 30697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.254189 30697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:32.254217 30697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.259923 30697 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:36427
I20260928 02:43:32.259961 30825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:36427 every 8 connection(s)
I20260928 02:43:32.260293 30697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:32.266873 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30697
I20260928 02:43:32.266954 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:32.267153 28086 external_mini_cluster.cc:1428] Setting key 33ba3a0e4e2bc1139c429ada50e18184
I20260928 02:43:32.269526 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.260976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:59397 (local address 127.27.109.190:35387)
0928 02:43:32.261050 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:32.261052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:32.261226 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:43:32.261743 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:32.261745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:32.261783 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:32.261839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:32.262813 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.263312 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.264307 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.264512 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.264597 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.266644 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:32.266646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:32.266646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:32.266653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:32.268202 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.268584 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.268585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.268586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.268639 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.268885 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.268886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.268887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.269001 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:32.269077 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:32.269393 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:43:32.269461 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:43:32.269824 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.260943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35387 (local address 127.27.109.130:59397)
0928 02:43:32.261109 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:32.261149 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:32.261633 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:32.261919 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:32.261929 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:32.262507 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:32.262703 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.262708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.263403 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.263404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.264181 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.264183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.264615 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.264616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.264739 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.265204 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:32.265212 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:32.266549 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:32.268279 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.268285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.268287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.268498 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.268712 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.268713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.268713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.268818 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.269068 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:32.269074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:32.269235 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 02:43:32.269362 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:43:32.269527 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260928 02:43:32.270298 30826 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35387
I20260928 02:43:32.270385 30826 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:32.270582 30826 heartbeater.cc:511] Master 127.27.109.190:35387 requested a full tablet report, sending...
I20260928 02:43:32.270938 30501 ts_manager.cc:194] Registered new tserver with Master: 97bb5deaaae1479dae9eb61f1d32b3ed (127.27.109.130:36427)
I20260928 02:43:32.271309 30501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:59397
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:32.290685 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:32.368696 30831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:32.368876 30831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:32.368898 30831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:32.370020 30831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:32.370061 30831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:32.370117 30831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:32.371536 30831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:32.371786 30831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:32.372025 30831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:32.376876 30839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:32.376893 30837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:32.377082 30836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:32.377144 30831 server_base.cc:1034] running on GCE node
I20260928 02:43:32.377337 30831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:32.377569 30831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:32.378705 30831 hybrid_clock.cc:648] HybridClock initialized: now 1790563412378696 us; error 30 us; skew 500 ppm
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:32.381598 30831 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:32.381991 30831 webserver.cc:460] Webserver started at http://127.27.109.131:39677/ using document root <none> and password file <none>
I20260928 02:43:32.382170 30831 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:32.382207 30831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:32.382302 30831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:32.383301 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "4e2f136b264e4e59b5d403e4c692cec6"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "69b64043b29417e86d2c50e572483460"
server_key_iv: "162a33c3c7572fec15d591178c6b1d71"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.383589 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "4e2f136b264e4e59b5d403e4c692cec6"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "69b64043b29417e86d2c50e572483460"
server_key_iv: "162a33c3c7572fec15d591178c6b1d71"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.384800 30831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:32.385407 30846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.385557 30831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:32.385639 30831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "4e2f136b264e4e59b5d403e4c692cec6"
format_stamp: "Formatted at 2026-09-28 02:43:32 on dist-test-slave-b5p5"
server_key: "69b64043b29417e86d2c50e572483460"
server_key_iv: "162a33c3c7572fec15d591178c6b1d71"
server_key_version: "encryptionkey@0"
I20260928 02:43:32.385690 30831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:32.407792 30831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:32.411166 30831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:32.411321 30831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:32.411744 30831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:32.412089 30831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:32.412153 30831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.412186 30831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:32.412214 30831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:32.420396 30831 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:32833
I20260928 02:43:32.420835 30831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:32.422626 30959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:32833 every 8 connection(s)
I20260928 02:43:32.424973 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30831
I20260928 02:43:32.425071 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:32.425256 28086 external_mini_cluster.cc:1428] Setting key 439c6a6998be3dc247067acf58621e4a
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:32.432567 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.421656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35387 (local address 127.27.109.131:57021)
0928 02:43:32.421838 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:32.421882 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:32.422739 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:32.422997 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:32.423001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:32.423639 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:32.423885 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.423891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.424819 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.424822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.426052 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:32.426056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.426651 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.426651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:32.426852 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.427375 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:32.427386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:32.429250 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:32.431118 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.431124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.431126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.431403 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.431609 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:32.431609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:32.431610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:32.431721 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:32.432018 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:32.432024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:32.432186 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 02:43:32.432295 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:43:32.432413 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260928 02:43:32.432565 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:32.421966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:57021 (local address 127.27.109.190:35387)
0928 02:43:32.422037 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:32.422040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:32.422053 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:32.422836 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:32.422838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:32.422852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:32.422913 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:32.423978 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.424712 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.426156 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:32.426504 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:32.426560 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:32.429377 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:32.429378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:32.429379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:32.429387 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:32.431023 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.431479 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.431480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.431480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.431544 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:32.431783 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:32.431784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:32.431784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:32.431936 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:32.431992 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:32.432345 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:43:32.432434 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:43:32.433064 30960 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35387
I20260928 02:43:32.433187 30960 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:32.433379 30960 heartbeater.cc:511] Master 127.27.109.190:35387 requested a full tablet report, sending...
I20260928 02:43:32.433743 30501 ts_manager.cc:194] Registered new tserver with Master: 4e2f136b264e4e59b5d403e4c692cec6 (127.27.109.131:32833)
I20260928 02:43:32.434116 30501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:57021
I20260928 02:43:32.437055 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:43:32.447793 30541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:32.446847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.27.109.190:35387)
0928 02:43:32.446923 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:32.446925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:32.447023 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:43:32.447113 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:32.447114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:32.447125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:32.447194 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:32.447698 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36804: BlockingRecv error: recv got EOF from 127.0.0.1:36804 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:43:32.448638 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30563
I20260928 02:43:32.452701 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30697
I20260928 02:43:32.456842 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30831
I20260928 02:43:32.460772 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30469
2026-09-28T02:43:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1825 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30972](info): set up 2 sockets
Sep 28 02:43:32 dist-test-slave-b5p5 krb5kdc[30972](info): commencing operation
krb5kdc: starting...
W20260928 02:43:33.494689 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[30972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:33.610806 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39389
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:33.688714 30985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:33.688949 30985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:33.688985 30985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:33.690217 30985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:33.690276 30985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:33.690300 30985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:33.690321 30985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:33.690351 30985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:33.691823 30985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:33.692085 30985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:33.692301 30985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:43:33.697081 30985 server_base.cc:1034] running on GCE node
W20260928 02:43:33.697016 30990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:33.697057 30993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:33.697014 30991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:33.697433 30985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:33.697745 30985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:33.698935 30985 hybrid_clock.cc:648] HybridClock initialized: now 1790563413698912 us; error 34 us; skew 500 ppm
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[30972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:33.701304 30985 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:33.701723 30985 webserver.cc:460] Webserver started at http://127.27.109.190:46791/ using document root <none> and password file <none>
I20260928 02:43:33.701960 30985 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:33.702028 30985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:33.702174 30985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:33.703265 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "4399301689dc4de7b7e4d5545c3b6255"
format_stamp: "Formatted at 2026-09-28 02:43:33 on dist-test-slave-b5p5"
server_key: "ff298a5d7d966cff2537d76ff40e7690"
server_key_iv: "b00369d2bbb50a1e5144b4036dec5b88"
server_key_version: "encryptionkey@0"
I20260928 02:43:33.703619 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "4399301689dc4de7b7e4d5545c3b6255"
format_stamp: "Formatted at 2026-09-28 02:43:33 on dist-test-slave-b5p5"
server_key: "ff298a5d7d966cff2537d76ff40e7690"
server_key_iv: "b00369d2bbb50a1e5144b4036dec5b88"
server_key_version: "encryptionkey@0"
I20260928 02:43:33.704988 30985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:43:33.705641 31000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:33.705843 30985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:33.705933 30985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "4399301689dc4de7b7e4d5545c3b6255"
format_stamp: "Formatted at 2026-09-28 02:43:33 on dist-test-slave-b5p5"
server_key: "ff298a5d7d966cff2537d76ff40e7690"
server_key_iv: "b00369d2bbb50a1e5144b4036dec5b88"
server_key_version: "encryptionkey@0"
I20260928 02:43:33.706015 30985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:33.724406 30985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:33.726339 30985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:33.726552 30985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:33.730260 30985 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39389
I20260928 02:43:33.730309 31052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39389 every 8 connection(s)
I20260928 02:43:33.730813 30985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:33.731513 31053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:33.733744 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255: Bootstrap starting.
I20260928 02:43:33.734345 31053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:33.734668 31053 log.cc:824] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:33.734956 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 30985
I20260928 02:43:33.735039 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:33.735237 28086 external_mini_cluster.cc:1428] Setting key d503a07757bc46d50f1dfd45de245cba
I20260928 02:43:33.735423 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255: No bootstrap required, opened a new log
I20260928 02:43:33.738147 31053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } }
I20260928 02:43:33.738310 31053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:33.738343 31053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4399301689dc4de7b7e4d5545c3b6255, State: Initialized, Role: FOLLOWER
I20260928 02:43:33.738456 31053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } }
I20260928 02:43:33.738554 31053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:33.738582 31053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:33.738626 31053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:33.739271 31053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } }
I20260928 02:43:33.739359 31053 leader_election.cc:302] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4399301689dc4de7b7e4d5545c3b6255; no voters:
I20260928 02:43:33.739530 31053 leader_election.cc:288] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:33.739581 31058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:33.739701 31058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [term 1 LEADER]: Becoming Leader. State: Replica: 4399301689dc4de7b7e4d5545c3b6255, State: Running, Role: LEADER
I20260928 02:43:33.739799 31053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:33.739780 31058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } }
I20260928 02:43:33.740218 31060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4399301689dc4de7b7e4d5545c3b6255. Latest consensus state: current_term: 1 leader_uuid: "4399301689dc4de7b7e4d5545c3b6255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } } }
I20260928 02:43:33.740455 31060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:33.740379 31059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4399301689dc4de7b7e4d5545c3b6255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399301689dc4de7b7e4d5545c3b6255" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39389 } } }
I20260928 02:43:33.740731 31059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:33.740825 31066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:33.741336 31066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:33.742916 31066 catalog_manager.cc:1347] Generated new cluster ID: e48a14a64b144e7799870dbf665df6d9
I20260928 02:43:33.742970 31066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[30972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:33.746822 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:33.735705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.27.109.190:39389)
0928 02:43:33.735899 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:33.735909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:33.735946 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:43:33.736768 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:33.736777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:33.736841 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:33.736991 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:33.737578 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:33.738996 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:33.740043 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:33.740660 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:33.740757 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:33.743615 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:33.743618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:33.743618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:33.743629 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:33.745351 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:33.745714 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:33.745716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:33.745717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:33.745810 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:33.746020 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:33.746020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:33.746021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:33.746286 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:33.746413 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:33.746505 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:43:33.746622 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 02:43:33.747735 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:33.766168 31066 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:33.766924 31066 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:33.767400 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:33.763420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.27.109.190:39389)
0928 02:43:33.763463 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:33.763465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:33.763475 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:33.763499 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:33.763500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:33.763509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:33.763561 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:33.763957 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:33.764784 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:33.765768 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:33.766030 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:33.766115 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:33.766911 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:33.766913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:33.766913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:33.767122 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:33.767140 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:33.767147 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:33.767211 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:33.767282 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:43:33.767324 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:43:33.776083 31066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4399301689dc4de7b7e4d5545c3b6255: Generated new TSK 0
I20260928 02:43:33.776260 31066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:33.784327 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 30985
2026-09-28T02:43:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1321 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[31086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[31086](info): set up 2 sockets
Sep 28 02:43:33 dist-test-slave-b5p5 krb5kdc[31086](info): commencing operation
krb5kdc: starting...
W20260928 02:43:34.806649 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:34 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:34.916810 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37925
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:34.995814 31099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:34.996062 31099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:34.996100 31099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:34.997253 31099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:34.997326 31099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:34.997350 31099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:34.997380 31099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:34.997409 31099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:34.998896 31099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:34.999172 31099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:34.999406 31099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:35.004371 31104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.004390 31105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.004390 31107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:35.004510 31099 server_base.cc:1034] running on GCE node
I20260928 02:43:35.004806 31099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:35.005059 31099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:35.006222 31099 hybrid_clock.cc:648] HybridClock initialized: now 1790563415006208 us; error 26 us; skew 500 ppm
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:35.008738 31099 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:35.009261 31099 webserver.cc:460] Webserver started at http://127.27.109.190:44195/ using document root <none> and password file <none>
I20260928 02:43:35.009505 31099 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:35.009574 31099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:35.009732 31099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:35.010860 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "c5d9d52c9a694e5fabb381b4b63e7d16"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "c0ff5b0b8bb47a2d78b63ca81d3ebd90"
server_key_iv: "9ebd731cd4130b4c7428bf2b710ed1fd"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.011219 31099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "c5d9d52c9a694e5fabb381b4b63e7d16"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "c0ff5b0b8bb47a2d78b63ca81d3ebd90"
server_key_iv: "9ebd731cd4130b4c7428bf2b710ed1fd"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.012519 31099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:35.013258 31114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.013420 31099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:35.013520 31099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "c5d9d52c9a694e5fabb381b4b63e7d16"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "c0ff5b0b8bb47a2d78b63ca81d3ebd90"
server_key_iv: "9ebd731cd4130b4c7428bf2b710ed1fd"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.013600 31099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:35.028637 31099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:35.030588 31099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:35.030757 31099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:35.035094 31099 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37925
I20260928 02:43:35.035142 31166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37925 every 8 connection(s)
I20260928 02:43:35.035521 31099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:35.036528 31167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:35.038844 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16: Bootstrap starting.
I20260928 02:43:35.039371 31167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:35.039593 31167 log.cc:824] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:35.040230 31167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16: No bootstrap required, opened a new log
I20260928 02:43:35.041069 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31099
I20260928 02:43:35.041167 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:35.041358 28086 external_mini_cluster.cc:1428] Setting key ead57121a19e5007529c1682371497ba
I20260928 02:43:35.042332 31167 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } }
I20260928 02:43:35.042524 31167 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:35.042575 31167 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d9d52c9a694e5fabb381b4b63e7d16, State: Initialized, Role: FOLLOWER
I20260928 02:43:35.042713 31167 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } }
I20260928 02:43:35.042805 31167 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:35.042847 31167 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:35.042900 31167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:35.043840 31167 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } }
I20260928 02:43:35.043926 31167 leader_election.cc:302] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5d9d52c9a694e5fabb381b4b63e7d16; no voters:
I20260928 02:43:35.044117 31167 leader_election.cc:288] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:35.044137 31172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:35.044245 31172 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [term 1 LEADER]: Becoming Leader. State: Replica: c5d9d52c9a694e5fabb381b4b63e7d16, State: Running, Role: LEADER
I20260928 02:43:35.044324 31172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } }
I20260928 02:43:35.044374 31167 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:35.044672 31172 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5d9d52c9a694e5fabb381b4b63e7d16. Latest consensus state: current_term: 1 leader_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } } }
I20260928 02:43:35.044750 31172 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:35.044684 31173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5d9d52c9a694e5fabb381b4b63e7d16" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37925 } } }
I20260928 02:43:35.044899 31173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:35.045254 31177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:35.045799 31177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:35.048979 31177 catalog_manager.cc:1347] Generated new cluster ID: ce255b22d62749739fc7f4f562e8d1f2
I20260928 02:43:35.049042 31177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:35.052261 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.041909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.27.109.190:37925)
0928 02:43:35.042089 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:35.042096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:35.042121 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:43:35.042309 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:35.042314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:35.042358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:35.042504 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:35.043309 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.044360 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.045364 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.045979 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.046125 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.048953 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:35.048957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:35.048957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:35.048968 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:35.050709 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.051083 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.051086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.051086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.051199 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.051422 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.051423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.051423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.051661 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:35.051828 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:35.051931 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:43:35.052071 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260928 02:43:35.056011 31177 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:35.056692 31177 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:35.060223 31177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5d9d52c9a694e5fabb381b4b63e7d16: Generated new TSK 0
I20260928 02:43:35.060392 31177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:35.089519 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:35.171528 31193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:35.171772 31193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:35.171818 31193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:35.172956 31193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:35.173025 31193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:35.173067 31193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:35.174492 31193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:35.174753 31193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:35.174988 31193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:35.180085 31198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.180123 31199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.180100 31201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:35.180332 31193 server_base.cc:1034] running on GCE node
I20260928 02:43:35.180575 31193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:35.180800 31193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:35.181959 31193 hybrid_clock.cc:648] HybridClock initialized: now 1790563415181954 us; error 37 us; skew 500 ppm
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:35.184350 31193 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:35.184764 31193 webserver.cc:460] Webserver started at http://127.27.109.129:39239/ using document root <none> and password file <none>
I20260928 02:43:35.184986 31193 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:35.185050 31193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:35.185190 31193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:35.186211 31193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "9ff7350f65d349e9ad077c93a4121bde"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "b0c2db4f60471745e5ccc80716dbc17c"
server_key_iv: "ba346b85e7481925e4df9d29c2a7704a"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.186611 31193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "9ff7350f65d349e9ad077c93a4121bde"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "b0c2db4f60471745e5ccc80716dbc17c"
server_key_iv: "ba346b85e7481925e4df9d29c2a7704a"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.187872 31193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:35.188630 31208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.188808 31193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:35.188903 31193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9ff7350f65d349e9ad077c93a4121bde"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "b0c2db4f60471745e5ccc80716dbc17c"
server_key_iv: "ba346b85e7481925e4df9d29c2a7704a"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.188982 31193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:35.208069 31193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:35.210582 31193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:35.210736 31193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:35.210981 31193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:35.211289 31193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:35.211341 31193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.211376 31193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:35.211402 31193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.217288 31193 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37043
I20260928 02:43:35.217334 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37043 every 8 connection(s)
I20260928 02:43:35.217698 31193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:35.224870 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31193
I20260928 02:43:35.224982 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:35.225224 28086 external_mini_cluster.cc:1428] Setting key 9ae8f1654a6d3d6fcfe6e22d3cf1eb56
I20260928 02:43:35.227555 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.218422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:44981 (local address 127.27.109.190:37925)
0928 02:43:35.218517 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:35.218520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:35.218698 (+ 178us) server_negotiation.cc:407] Connection header received
0928 02:43:35.219238 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:35.219239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:35.219254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:35.219306 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:35.220233 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.220733 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.221746 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.221955 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.222060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.224260 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:35.224262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:35.224263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:35.224270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:35.225836 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.226392 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.226393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.226393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.226450 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.226788 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.226789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.226789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.226911 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:35.226965 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:35.227320 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 02:43:35.227439 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:43:35.227706 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.218392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37925 (local address 127.27.109.129:44981)
0928 02:43:35.218583 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:35.218617 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:35.219124 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:35.219388 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:35.219392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:35.219946 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:35.220133 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.220137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.220837 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.220838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.221645 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.221647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.222065 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.222066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.222190 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.222741 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:35.222750 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:35.224155 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:35.225971 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.225978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.225980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.226303 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.226573 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.226574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.226574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.226722 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.226978 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:35.226983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:35.227152 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 02:43:35.227266 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:43:35.227447 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260928 02:43:35.228168 31322 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37925
I20260928 02:43:35.228276 31322 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:35.228482 31322 heartbeater.cc:511] Master 127.27.109.190:37925 requested a full tablet report, sending...
I20260928 02:43:35.228986 31131 ts_manager.cc:194] Registered new tserver with Master: 9ff7350f65d349e9ad077c93a4121bde (127.27.109.129:37043)
I20260928 02:43:35.229679 31131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:44981
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:35.249939 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:35.329209 31327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:35.329443 31327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:35.329480 31327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:35.330675 31327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:35.330734 31327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:35.330776 31327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:35.332207 31327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:35.332472 31327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:35.332711 31327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:35.337711 31335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.337697 31332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.337697 31333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:35.337962 31327 server_base.cc:1034] running on GCE node
I20260928 02:43:35.338150 31327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:35.338410 31327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:35.339594 31327 hybrid_clock.cc:648] HybridClock initialized: now 1790563415339579 us; error 30 us; skew 500 ppm
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:35.342034 31327 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:35.342506 31327 webserver.cc:460] Webserver started at http://127.27.109.130:34717/ using document root <none> and password file <none>
I20260928 02:43:35.342751 31327 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:35.342816 31327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:35.342960 31327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:35.344163 31327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "461b57945a10475499e1528eee939507"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "12df55e78b4a736eb28460099de09bd9"
server_key_iv: "61310cc8962de874272e0b4d43d2fc76"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.344543 31327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "461b57945a10475499e1528eee939507"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "12df55e78b4a736eb28460099de09bd9"
server_key_iv: "61310cc8962de874272e0b4d43d2fc76"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.346040 31327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:35.346834 31342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.347023 31327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.347126 31327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "461b57945a10475499e1528eee939507"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "12df55e78b4a736eb28460099de09bd9"
server_key_iv: "61310cc8962de874272e0b4d43d2fc76"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.347206 31327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:35.364111 31327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:35.366575 31327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:35.366743 31327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:35.366992 31327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:35.367311 31327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:35.367367 31327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.367408 31327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:35.367436 31327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.373055 31327 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:42785
I20260928 02:43:35.373099 31455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:42785 every 8 connection(s)
I20260928 02:43:35.373445 31327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:35.374480 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31327
I20260928 02:43:35.374595 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:35.374797 28086 external_mini_cluster.cc:1428] Setting key 38f57fcda160594498ae4a23b7cab1f3
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:43:35.383376 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.374135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:60761 (local address 127.27.109.190:37925)
0928 02:43:35.374200 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:35.374202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:35.374495 (+ 293us) server_negotiation.cc:407] Connection header received
0928 02:43:35.375199 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:35.375200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:35.375209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:35.375236 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:35.376505 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.376983 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.378035 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.378381 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.378457 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.380460 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:35.380461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:35.380462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:35.380470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:35.382017 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.382379 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.382380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.382381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.382441 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.382708 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.382709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.382710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.382850 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:35.382928 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:35.383152 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:43:35.383236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:35.383414 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.374125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37925 (local address 127.27.109.130:60761)
0928 02:43:35.374312 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:35.374361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:35.375088 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:35.375315 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:35.375319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:35.376136 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:35.376411 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.376417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.377078 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.377079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.377940 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.377943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.378497 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.378499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.378617 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.379097 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:35.379104 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:35.380373 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:35.382104 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.382110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.382111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.382306 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.382549 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.382549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.382549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.382639 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.382922 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:35.382926 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:35.383030 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:43:35.383108 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:43:35.383216 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260928 02:43:35.383881 31456 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37925
I20260928 02:43:35.383978 31456 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:35.384150 31456 heartbeater.cc:511] Master 127.27.109.190:37925 requested a full tablet report, sending...
I20260928 02:43:35.384543 31131 ts_manager.cc:194] Registered new tserver with Master: 461b57945a10475499e1528eee939507 (127.27.109.130:42785)
I20260928 02:43:35.384938 31131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:60761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:35.399422 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:35.481842 31461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:35.482026 31461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:35.482048 31461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:35.483186 31461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:35.483222 31461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:35.483247 31461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:35.484592 31461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:35.484788 31461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:35.484951 31461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:35.489732 31466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.489790 31469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:35.489761 31467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:35.489882 31461 server_base.cc:1034] running on GCE node
I20260928 02:43:35.490110 31461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:35.490305 31461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:35.491469 31461 hybrid_clock.cc:648] HybridClock initialized: now 1790563415491458 us; error 28 us; skew 500 ppm
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:35.494082 31461 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:35.494457 31461 webserver.cc:460] Webserver started at http://127.27.109.131:36557/ using document root <none> and password file <none>
I20260928 02:43:35.494709 31461 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:35.494751 31461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:35.494835 31461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:35.495797 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "f839a294ab864813821220bb42bf3bb8"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "6952fa8aa97588d1c2e4e66e6b79b658"
server_key_iv: "16b7fedaec86ff7f118b6156c7f29ef4"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.496088 31461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "f839a294ab864813821220bb42bf3bb8"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "6952fa8aa97588d1c2e4e66e6b79b658"
server_key_iv: "16b7fedaec86ff7f118b6156c7f29ef4"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.497334 31461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:35.497993 31476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.498138 31461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:35.498215 31461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "f839a294ab864813821220bb42bf3bb8"
format_stamp: "Formatted at 2026-09-28 02:43:35 on dist-test-slave-b5p5"
server_key: "6952fa8aa97588d1c2e4e66e6b79b658"
server_key_iv: "16b7fedaec86ff7f118b6156c7f29ef4"
server_key_version: "encryptionkey@0"
I20260928 02:43:35.498266 31461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:35.509508 31461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:35.511417 31461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:35.511518 31461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:35.511669 31461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:35.511901 31461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:35.511930 31461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.511950 31461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:35.511962 31461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:35.517992 31461 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:45575
I20260928 02:43:35.518041 31589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:45575 every 8 connection(s)
I20260928 02:43:35.518342 31461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:35.523627 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31461
I20260928 02:43:35.523736 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:35.523950 28086 external_mini_cluster.cc:1428] Setting key 4378d0a0835fa2fbe8cecc4441539c72
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:35.527486 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.519068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:60803 (local address 127.27.109.190:37925)
0928 02:43:35.519111 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:35.519114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:35.519301 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:43:35.519752 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:35.519753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:35.519763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:35.519814 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:35.520648 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.521150 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.522078 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.522263 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.522319 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.524662 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:35.524663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:35.524664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:35.524671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:35.526078 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.526547 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.526548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.526548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.526606 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.526868 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.526869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.526870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.526977 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:35.527049 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:35.527323 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:43:35.527398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:35.527659 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.519055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37925 (local address 127.27.109.131:60803)
0928 02:43:35.519187 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:35.519218 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:35.519648 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:35.519865 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:35.519868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:35.520396 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:35.520564 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.520568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.521227 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.521228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.521999 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:35.522001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.522336 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.522336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:35.522427 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.522963 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:35.522971 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:35.524549 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:35.526198 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.526204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.526212 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.526424 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.526675 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:35.526676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:35.526676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:35.526785 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:35.527055 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:35.527061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:35.527188 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 02:43:35.527300 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:43:35.527446 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260928 02:43:35.528123 31590 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37925
I20260928 02:43:35.528194 31590 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:35.528379 31590 heartbeater.cc:511] Master 127.27.109.190:37925 requested a full tablet report, sending...
I20260928 02:43:35.528753 31131 ts_manager.cc:194] Registered new tserver with Master: f839a294ab864813821220bb42bf3bb8 (127.27.109.131:45575)
I20260928 02:43:35.529140 31131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:60803
I20260928 02:43:35.535539 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563415, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:35.557473 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:35.550438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.27.109.190:37925)
0928 02:43:35.550505 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:35.550508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:35.550599 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:43:35.550648 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:35.550649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:35.550657 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:35.550712 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:35.551287 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.551756 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.552440 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:35.552617 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:35.552668 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:35.554795 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:35.554796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:35.554797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:35.554803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:35.556321 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.556676 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.556677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.556678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.556732 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:35.556949 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:35.556950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:35.556950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:35.557091 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:35.557190 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:35.557297 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:43:35.557372 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 02:43:35.557669 31131 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46330
I20260928 02:43:35.558022 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31193
I20260928 02:43:35.562942 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31327
I20260928 02:43:35.567273 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31461
I20260928 02:43:35.571255 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31099
2026-09-28T02:43:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1788 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31602](info): set up 2 sockets
Sep 28 02:43:35 dist-test-slave-b5p5 krb5kdc[31602](info): commencing operation
krb5kdc: starting...
W20260928 02:43:36.614830 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:36 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:36.729238 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39391
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:36.807489 31615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:36.807682 31615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:36.807705 31615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:36.808845 31615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:36.808895 31615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:36.808910 31615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:36.808923 31615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:36.808935 31615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:36.810396 31615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:36.810640 31615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:36.810814 31615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:36.815512 31623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:36.815517 31620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:36.815733 31615 server_base.cc:1034] running on GCE node
W20260928 02:43:36.815815 31621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:36.816068 31615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:36.816355 31615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:36.817526 31615 hybrid_clock.cc:648] HybridClock initialized: now 1790563416817514 us; error 45 us; skew 500 ppm
Sep 28 02:43:36 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:36.819995 31615 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:36.820411 31615 webserver.cc:460] Webserver started at http://127.27.109.190:39847/ using document root <none> and password file <none>
I20260928 02:43:36.820627 31615 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:36.820693 31615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:36.820843 31615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:36.821844 31615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "79d7d7bdd6d744648efed8d21f740941"
format_stamp: "Formatted at 2026-09-28 02:43:36 on dist-test-slave-b5p5"
server_key: "484493f4e75e428abe65fee38823c6d3"
server_key_iv: "6efe57db2d192fad3f1e5ced955b4ebe"
server_key_version: "encryptionkey@0"
I20260928 02:43:36.822189 31615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "79d7d7bdd6d744648efed8d21f740941"
format_stamp: "Formatted at 2026-09-28 02:43:36 on dist-test-slave-b5p5"
server_key: "484493f4e75e428abe65fee38823c6d3"
server_key_iv: "6efe57db2d192fad3f1e5ced955b4ebe"
server_key_version: "encryptionkey@0"
I20260928 02:43:36.823598 31615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:36.824283 31630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:36.824440 31615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:36.824535 31615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "79d7d7bdd6d744648efed8d21f740941"
format_stamp: "Formatted at 2026-09-28 02:43:36 on dist-test-slave-b5p5"
server_key: "484493f4e75e428abe65fee38823c6d3"
server_key_iv: "6efe57db2d192fad3f1e5ced955b4ebe"
server_key_version: "encryptionkey@0"
I20260928 02:43:36.824615 31615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:36.874547 31615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:36.876996 31615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:36.877166 31615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:36.881035 31615 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39391
I20260928 02:43:36.881073 31682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39391 every 8 connection(s)
I20260928 02:43:36.881603 31615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:36.882268 31683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:36.883826 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31615
I20260928 02:43:36.883912 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:36.884097 28086 external_mini_cluster.cc:1428] Setting key 626eb9decd7468a0944fd4c9a209ecf9
I20260928 02:43:36.884856 31683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: Bootstrap starting.
I20260928 02:43:36.885545 31683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:36.885871 31683 log.cc:824] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:36.886596 31683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: No bootstrap required, opened a new log
I20260928 02:43:36.888847 31683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } }
I20260928 02:43:36.889024 31683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:36.889067 31683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d7d7bdd6d744648efed8d21f740941, State: Initialized, Role: FOLLOWER
I20260928 02:43:36.889231 31683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } }
I20260928 02:43:36.889336 31683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:36.889396 31683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:36.889447 31683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:36.889943 31683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } }
I20260928 02:43:36.890058 31683 leader_election.cc:302] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d7d7bdd6d744648efed8d21f740941; no voters:
I20260928 02:43:36.890225 31683 leader_election.cc:288] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:36.890264 31688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:36.890385 31688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [term 1 LEADER]: Becoming Leader. State: Replica: 79d7d7bdd6d744648efed8d21f740941, State: Running, Role: LEADER
I20260928 02:43:36.890501 31683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:36.890564 31688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } }
I20260928 02:43:36.891033 31689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d7d7bdd6d744648efed8d21f740941. Latest consensus state: current_term: 1 leader_uuid: "79d7d7bdd6d744648efed8d21f740941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } } }
I20260928 02:43:36.891045 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d7d7bdd6d744648efed8d21f740941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d7d7bdd6d744648efed8d21f740941" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39391 } } }
I20260928 02:43:36.891207 31689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:36.891242 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941 [sys.catalog]: This master's current role is: LEADER
Sep 28 02:43:36 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
W20260928 02:43:36.893392 31704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:36.893448 31704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:36.893503 31692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:36.893653 31692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:36.895157 31692 catalog_manager.cc:1347] Generated new cluster ID: 37b283208de8481d99f6312699059e04
I20260928 02:43:36.895208 31692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:36.895773 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:36.884623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.27.109.190:39391)
0928 02:43:36.884824 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:36.884833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:36.884865 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:43:36.885450 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:36.885459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:36.885522 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:36.885680 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:36.886329 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:36.887700 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:36.888419 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:36.889030 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:36.889139 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:36.891829 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:36.891834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:36.891834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:36.891845 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:36.893769 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:36.894168 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:36.894172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:36.894172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:36.894295 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:36.894658 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:36.894659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:36.894659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:36.894934 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:36.895076 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:36.895520 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:43:36.895636 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260928 02:43:36.902408 31692 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:36.902917 31692 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:36.912142 31692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d7d7bdd6d744648efed8d21f740941: Generated new TSK 0
I20260928 02:43:36.912314 31692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:36.941783 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:37.020473 31709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:37.020701 31709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:37.020741 31709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:37.021908 31709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:37.021971 31709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:37.022017 31709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:37.023564 31709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:37.023808 31709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:37.024031 31709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:37.028836 31717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:37.028853 31715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:37.028853 31714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:37.029089 31709 server_base.cc:1034] running on GCE node
I20260928 02:43:37.029299 31709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:37.029541 31709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:37.030696 31709 hybrid_clock.cc:648] HybridClock initialized: now 1790563417030690 us; error 34 us; skew 500 ppm
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:37.033121 31709 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:37.033514 31709 webserver.cc:460] Webserver started at http://127.27.109.129:38739/ using document root <none> and password file <none>
I20260928 02:43:37.033739 31709 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:37.033824 31709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:37.033968 31709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:37.035000 31709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "f3694b024d4b4d8f8d5488516613dcfb"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "29ae05a5f8cb34abd35210f44c54f4ea"
server_key_iv: "05ae43b9e4fcf3f086c6297ef6904891"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.035344 31709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "f3694b024d4b4d8f8d5488516613dcfb"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "29ae05a5f8cb34abd35210f44c54f4ea"
server_key_iv: "05ae43b9e4fcf3f086c6297ef6904891"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.036635 31709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:37.037297 31724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.037460 31709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:37.037557 31709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "f3694b024d4b4d8f8d5488516613dcfb"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "29ae05a5f8cb34abd35210f44c54f4ea"
server_key_iv: "05ae43b9e4fcf3f086c6297ef6904891"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.037632 31709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:37.044588 31709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:37.046838 31709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:37.046991 31709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:37.047204 31709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:37.047492 31709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:37.047544 31709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.047583 31709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:37.047611 31709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.053078 31709 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:36977
I20260928 02:43:37.053144 31837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:36977 every 8 connection(s)
I20260928 02:43:37.053460 31709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:37.055917 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31709
I20260928 02:43:37.056030 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:37.056231 28086 external_mini_cluster.cc:1428] Setting key 03842f8fd2e11e81f9783ade667edec0
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:37.064308 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.054157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39391 (local address 127.27.109.129:50767)
0928 02:43:37.054320 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:37.054352 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:37.055267 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:37.055530 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:37.055534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:37.055820 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:37.056111 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.056119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.056988 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.056989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.058325 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.058329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.058857 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.058857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.058963 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.059511 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:37.059529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:37.061066 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:37.063052 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.063057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.063059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.063271 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.063485 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.063486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.063486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.063573 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.063864 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:37.063868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:37.063967 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:43:37.064043 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:43:37.064151 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 02:43:37.064307 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.054188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:50767 (local address 127.27.109.190:39391)
0928 02:43:37.054271 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:37.054273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:37.054431 (+ 158us) server_negotiation.cc:407] Connection header received
0928 02:43:37.055395 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:37.055396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:37.055409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:37.055459 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:37.056220 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.056884 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.058424 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.058774 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.058849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.061158 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:37.061160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:37.061160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:37.061168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:37.062970 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.063350 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.063351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.063352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.063417 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.063638 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.063639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.063640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.063795 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:37.063872 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:37.064096 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:43:37.064196 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:43:37.064803 31838 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39391
I20260928 02:43:37.064904 31838 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:37.065104 31838 heartbeater.cc:511] Master 127.27.109.190:39391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
I20260928 02:43:37.065634 31647 ts_manager.cc:194] Registered new tserver with Master: f3694b024d4b4d8f8d5488516613dcfb (127.27.109.129:36977)
I20260928 02:43:37.066239 31647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:50767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:37.081460 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:37.165002 31843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:37.165297 31843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:37.165344 31843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:37.166608 31843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:37.166677 31843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:37.166723 31843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:37.168198 31843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:37.168450 31843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:37.168669 31843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:37.173465 31848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:37.173533 31843 server_base.cc:1034] running on GCE node
W20260928 02:43:37.173480 31849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:37.173465 31851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:37.173879 31843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:37.174067 31843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:37.175209 31843 hybrid_clock.cc:648] HybridClock initialized: now 1790563417175197 us; error 27 us; skew 500 ppm
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:37.177557 31843 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:37.178035 31843 webserver.cc:460] Webserver started at http://127.27.109.130:38443/ using document root <none> and password file <none>
I20260928 02:43:37.178252 31843 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:37.178295 31843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:37.178453 31843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:37.179441 31843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "bffecab7d3ca4e0db3a7b4a23838908a"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "3cd6f821b9c84d6040d86a831f2dd068"
server_key_iv: "cbda5b623de70792127b5f0536f2a706"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.179819 31843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "bffecab7d3ca4e0db3a7b4a23838908a"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "3cd6f821b9c84d6040d86a831f2dd068"
server_key_iv: "cbda5b623de70792127b5f0536f2a706"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.181066 31843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:43:37.181818 31858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.182000 31843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:37.182097 31843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "bffecab7d3ca4e0db3a7b4a23838908a"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "3cd6f821b9c84d6040d86a831f2dd068"
server_key_iv: "cbda5b623de70792127b5f0536f2a706"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.182186 31843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:37.201400 31843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:37.203234 31843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:37.203389 31843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:37.203596 31843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:37.203886 31843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:37.203941 31843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.203981 31843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:37.204008 31843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.209749 31843 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44077
I20260928 02:43:37.209806 31971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44077 every 8 connection(s)
I20260928 02:43:37.210117 31843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:37.215893 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31843
I20260928 02:43:37.216012 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:37.216187 28086 external_mini_cluster.cc:1428] Setting key 16fcd20b93e2674a6af240a93507fa42
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:37.219766 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.210771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39391 (local address 127.27.109.130:51629)
0928 02:43:37.210964 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:37.210997 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:37.211882 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:37.212151 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:37.212154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:37.212388 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:37.212592 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.212596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.213302 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.213303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.214085 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.214087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.214495 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.214496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.214619 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.215098 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:37.215106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:37.216632 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:37.218537 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.218542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.218544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.218761 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.218981 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.218982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.218982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.219075 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.219331 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:37.219335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:37.219435 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:43:37.219513 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:43:37.219618 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260928 02:43:37.219794 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.210861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:51629 (local address 127.27.109.190:39391)
0928 02:43:37.210930 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:37.210933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:37.211072 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:43:37.212001 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:37.212002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:37.212015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:37.212064 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:37.212687 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.213189 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.214184 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.214394 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.214472 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.216726 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:37.216727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:37.216727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:37.216735 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:37.218419 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.218842 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.218844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.218844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.218913 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.219124 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.219125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.219125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.219267 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:37.219337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:37.219559 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:43:37.219647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:37.220242 31972 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39391
I20260928 02:43:37.220340 31972 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:37.220537 31972 heartbeater.cc:511] Master 127.27.109.190:39391 requested a full tablet report, sending...
I20260928 02:43:37.220922 31647 ts_manager.cc:194] Registered new tserver with Master: bffecab7d3ca4e0db3a7b4a23838908a (127.27.109.130:44077)
I20260928 02:43:37.221290 31647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:51629
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:37.240579 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:37.319396 31977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:37.319623 31977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:37.319659 31977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:37.320797 31977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:37.320878 31977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:37.320923 31977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:37.322350 31977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:37.322633 31977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:37.322876 31977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:37.327551 31983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:37.327581 31982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:37.327551 31985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:37.328191 31977 server_base.cc:1034] running on GCE node
I20260928 02:43:37.328402 31977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:37.328629 31977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:37.329782 31977 hybrid_clock.cc:648] HybridClock initialized: now 1790563417329779 us; error 31 us; skew 500 ppm
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:37.332122 31977 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:37.332532 31977 webserver.cc:460] Webserver started at http://127.27.109.131:36543/ using document root <none> and password file <none>
I20260928 02:43:37.332702 31977 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:37.332741 31977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:37.332823 31977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:37.333813 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "2d47c917759b48548573108d624963a4"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "2d879642d8b7b203bd30b744da5ccef3"
server_key_iv: "65d350d601b883e62ce3ecefaf10cb5f"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.334111 31977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "2d47c917759b48548573108d624963a4"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "2d879642d8b7b203bd30b744da5ccef3"
server_key_iv: "65d350d601b883e62ce3ecefaf10cb5f"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.335464 31977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:37.336136 31992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.336333 31977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:37.336401 31977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "2d47c917759b48548573108d624963a4"
format_stamp: "Formatted at 2026-09-28 02:43:37 on dist-test-slave-b5p5"
server_key: "2d879642d8b7b203bd30b744da5ccef3"
server_key_iv: "65d350d601b883e62ce3ecefaf10cb5f"
server_key_version: "encryptionkey@0"
I20260928 02:43:37.336452 31977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:37.347177 31977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:37.348980 31977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:37.349086 31977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:37.349257 31977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:37.349488 31977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:37.349515 31977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.349536 31977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:37.349548 31977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:37.355150 31977 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:41909
I20260928 02:43:37.355211 32105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:41909 every 8 connection(s)
I20260928 02:43:37.355495 31977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:37.364058 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.356197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:55475 (local address 127.27.109.190:39391)
0928 02:43:37.356244 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:37.356246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:37.356445 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:43:37.357303 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:37.357304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:37.357314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:37.357368 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:37.357944 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.358436 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.359442 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.359619 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.359669 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.361727 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:37.361729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:37.361729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:37.361737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:37.362986 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.363334 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.363335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.363335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.363381 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.363577 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.363578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.363578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.363672 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:37.363720 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:37.363940 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:43:37.363995 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:37.364253 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.356166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39391 (local address 127.27.109.131:55475)
0928 02:43:37.356317 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:37.356349 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:37.357211 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:37.357422 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:37.357425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:37.357654 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:37.357871 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.357875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.358564 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.358566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.359367 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:37.359369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.359686 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.359687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:37.359807 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.360328 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:37.360336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:37.361644 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:37.363056 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.363062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.363063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.363271 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.363436 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:37.363436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:37.363437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:37.363525 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:37.363723 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:37.363727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:37.363835 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:43:37.363912 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:43:37.364020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 02:43:37.364775 32106 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39391
I20260928 02:43:37.364874 32106 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:37.365072 32106 heartbeater.cc:511] Master 127.27.109.190:39391 requested a full tablet report, sending...
I20260928 02:43:37.365105 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 31977
I20260928 02:43:37.365226 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:37.365432 28086 external_mini_cluster.cc:1428] Setting key 07adbc68f29d9829971a9d6ef076e4d9
I20260928 02:43:37.365464 31647 ts_manager.cc:194] Registered new tserver with Master: 2d47c917759b48548573108d624963a4 (127.27.109.131:41909)
I20260928 02:43:37.365836 31647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:55475
I20260928 02:43:37.366144 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:37.387811 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.380322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.27.109.129:36977)
0928 02:43:37.380517 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:37.380523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:37.380536 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:43:37.380569 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:37.380571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:37.380585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:37.380652 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:37.381177 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.381878 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.382585 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.382901 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.382964 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.385135 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:37.385143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:37.385143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:37.385157 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:37.386659 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.387001 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.387010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.387011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.387059 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.387258 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.387259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.387259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.387385 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:37.387538 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:37.387606 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:43:37.387678 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[31602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:37.395695 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:37.389037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.27.109.190:39391)
0928 02:43:37.389092 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:37.389094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:37.389118 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:43:37.389179 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:37.389180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:37.389190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:37.389229 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:37.389662 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.390128 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.390806 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:37.390983 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:37.391069 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:37.393228 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:37.393229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:37.393230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:37.393235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:37.394664 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.394951 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.394952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.394952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.394998 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:37.395194 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:37.395195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:37.395195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:37.395344 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:37.395451 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:37.395569 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:43:37.395634 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:43:37.395934 31646 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422
I20260928 02:43:37.396263 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31709
I20260928 02:43:37.401535 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31843
I20260928 02:43:37.405627 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31977
I20260928 02:43:37.411613 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 31615
2026-09-28T02:43:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1843 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[32118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[32118](info): set up 2 sockets
Sep 28 02:43:37 dist-test-slave-b5p5 krb5kdc[32118](info): commencing operation
krb5kdc: starting...
W20260928 02:43:38.446813 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:38.558807 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45641
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:45641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:38.637823 32131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:38.638033 32131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:38.638070 32131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:38.639222 32131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:38.639287 32131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:38.639312 32131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:38.639343 32131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:38.639370 32131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:38.640800 32131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:45641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:38.641048 32131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:38.641259 32131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:38.645902 32139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:38.645975 32131 server_base.cc:1034] running on GCE node
W20260928 02:43:38.645912 32137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:38.645912 32136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:38.646343 32131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:38.646641 32131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:38.647802 32131 hybrid_clock.cc:648] HybridClock initialized: now 1790563418647784 us; error 37 us; skew 500 ppm
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:38.650141 32131 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:38.650544 32131 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:38.650593 32131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:38.650681 32131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:38.651623 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "8f608ac611c84c64b8b8dae945acb568"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "32010ea2bfec861a66cb1382b8497bd4"
server_key_iv: "fbf176f7ffd29ff0e6cd515885345ef7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.651903 32131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "8f608ac611c84c64b8b8dae945acb568"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "32010ea2bfec861a66cb1382b8497bd4"
server_key_iv: "fbf176f7ffd29ff0e6cd515885345ef7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.653091 32131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:38.653745 32145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.653910 32131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:38.653995 32131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "8f608ac611c84c64b8b8dae945acb568"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "32010ea2bfec861a66cb1382b8497bd4"
server_key_iv: "fbf176f7ffd29ff0e6cd515885345ef7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.654058 32131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:38.673813 32131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:38.676081 32131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:38.676227 32131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:38.679854 32131 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:45641
I20260928 02:43:38.679883 32197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:45641 every 8 connection(s)
I20260928 02:43:38.680154 32131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:38.681058 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:38.682760 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32131
I20260928 02:43:38.682875 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:38.683064 28086 external_mini_cluster.cc:1428] Setting key 182b248895c6ac304ce139a8926351fe
I20260928 02:43:38.683434 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: Bootstrap starting.
I20260928 02:43:38.684108 32198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:38.684428 32198 log.cc:824] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:38.685187 32198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: No bootstrap required, opened a new log
I20260928 02:43:38.687175 32198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } }
I20260928 02:43:38.687361 32198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:38.687390 32198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f608ac611c84c64b8b8dae945acb568, State: Initialized, Role: FOLLOWER
I20260928 02:43:38.687538 32198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } }
I20260928 02:43:38.687603 32198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:38.687644 32198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:38.687688 32198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:38.688374 32198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } }
I20260928 02:43:38.688483 32198 leader_election.cc:302] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f608ac611c84c64b8b8dae945acb568; no voters:
I20260928 02:43:38.688647 32198 leader_election.cc:288] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:38.688712 32203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:38.688862 32203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [term 1 LEADER]: Becoming Leader. State: Replica: 8f608ac611c84c64b8b8dae945acb568, State: Running, Role: LEADER
I20260928 02:43:38.688912 32198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:38.689019 32203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } }
I20260928 02:43:38.689505 32205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f608ac611c84c64b8b8dae945acb568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } } }
I20260928 02:43:38.689517 32204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f608ac611c84c64b8b8dae945acb568. Latest consensus state: current_term: 1 leader_uuid: "8f608ac611c84c64b8b8dae945acb568" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f608ac611c84c64b8b8dae945acb568" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45641 } } }
I20260928 02:43:38.689634 32205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:38.689667 32204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568 [sys.catalog]: This master's current role is: LEADER
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:38.690388 32210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:38.690876 32210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:38.693883 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:38.683594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.27.109.190:45641)
0928 02:43:38.683797 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:38.683805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:38.683830 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:43:38.684273 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:38.684281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:38.684336 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:38.684443 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:38.685264 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.686150 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.687274 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.687866 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.687945 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:38.690742 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:38.690745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:38.690746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:38.690757 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:38.692412 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:38.692806 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:38.692808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:38.692809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:38.692884 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:38.693088 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:38.693089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:38.693089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:38.693300 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:38.693398 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:38.693540 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:43:38.693629 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260928 02:43:38.695403 32210 catalog_manager.cc:1347] Generated new cluster ID: 42e643b7ef634476a249ce657f623a96
I20260928 02:43:38.695464 32210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:38.703469 32210 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:38.704000 32210 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:38.709592 32210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: Generated new TSK 0
I20260928 02:43:38.709789 32210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:38.735316 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:38.814713 32224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:38.814899 32224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:38.814922 32224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:38.816052 32224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:38.816094 32224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:38.816120 32224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:38.817565 32224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:38.817790 32224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:38.818001 32224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:38.822816 32232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:38.822981 32224 server_base.cc:1034] running on GCE node
W20260928 02:43:38.822822 32229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:38.822847 32230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:38.823300 32224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:38.823544 32224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:38.824738 32224 hybrid_clock.cc:648] HybridClock initialized: now 1790563418824718 us; error 35 us; skew 500 ppm
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:38.827134 32224 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:38.827414 32224 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:38.827457 32224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:38.827543 32224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:38.828486 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "0e063afac3cd47bdb4acf452addf174d"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "3d2ca1d11731d1152fe79a7573ff1f25"
server_key_iv: "9c610f2f1cd062f9c1be9f39d6b1ee39"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.828779 32224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "0e063afac3cd47bdb4acf452addf174d"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "3d2ca1d11731d1152fe79a7573ff1f25"
server_key_iv: "9c610f2f1cd062f9c1be9f39d6b1ee39"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.830019 32224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:38.830710 32238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.830881 32224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:38.830952 32224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "0e063afac3cd47bdb4acf452addf174d"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "3d2ca1d11731d1152fe79a7573ff1f25"
server_key_iv: "9c610f2f1cd062f9c1be9f39d6b1ee39"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.831044 32224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:38.844587 32224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:38.846259 32224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:38.846421 32224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:38.846733 32224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:38.847044 32224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:38.847077 32224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.847100 32224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:38.847144 32224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.852632 32224 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:45251
I20260928 02:43:38.852697 32351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:45251 every 8 connection(s)
I20260928 02:43:38.852967 32224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:38.859606 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32224
I20260928 02:43:38.859735 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:38.859920 28086 external_mini_cluster.cc:1428] Setting key 17068bfb3d1bfb3f05cdb05f59d5350f
I20260928 02:43:38.862341 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:38.853636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:34175 (local address 127.27.109.190:45641)
0928 02:43:38.853695 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:38.853697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:38.853905 (+ 208us) server_negotiation.cc:407] Connection header received
0928 02:43:38.854411 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:38.854413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:38.854426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:38.854496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:38.855375 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.855886 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.856941 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.857163 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.857248 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:38.859314 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:38.859316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:38.859316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:38.859322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:38.860830 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:38.861314 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:38.861315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:38.861315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:38.861363 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:38.861627 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:38.861627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:38.861628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:38.861721 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:38.861797 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:38.862137 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 02:43:38.862224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:43:38.862353 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:38.853616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45641 (local address 127.27.109.129:34175)
0928 02:43:38.853772 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:38.853821 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:38.854294 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:38.854567 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:38.854572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:38.854813 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:38.855275 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:38.855279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.855986 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.855987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:38.856850 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:38.856853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:38.857256 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:38.857257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:38.857371 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:38.857916 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:38.857924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:38.859231 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:38.860917 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:38.860925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:38.860927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:38.861237 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:38.861428 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:38.861429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:38.861429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:38.861568 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:38.861806 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:38.861812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:38.861984 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 02:43:38.862094 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:43:38.862207 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 02:43:38.862882 32352 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45641
I20260928 02:43:38.862979 32352 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:38.863127 32352 heartbeater.cc:511] Master 127.27.109.190:45641 requested a full tablet report, sending...
I20260928 02:43:38.863598 32162 ts_manager.cc:194] Registered new tserver with Master: 0e063afac3cd47bdb4acf452addf174d (127.27.109.129:45251)
I20260928 02:43:38.864269 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:34175
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:38.884354 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:38.963058 32357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:38.963243 32357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:38.963267 32357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:38.964386 32357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:38.964425 32357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:38.964452 32357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:38.965936 32357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:38.966156 32357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:38.966384 32357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:38.971118 32365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:38.971118 32362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:38.971174 32363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:38.971376 32357 server_base.cc:1034] running on GCE node
I20260928 02:43:38.971571 32357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:38.971753 32357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:38.972896 32357 hybrid_clock.cc:648] HybridClock initialized: now 1790563418972872 us; error 38 us; skew 500 ppm
Sep 28 02:43:38 dist-test-slave-b5p5 krb5kdc[32118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:38.975297 32357 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:38.975627 32357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:38.975674 32357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:38.975759 32357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:38.976684 32357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "27e3d0d318a7440baa5f70966cd4236c"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "2b5ad78896be9ec4de2b86a42dd4df06"
server_key_iv: "31cdd8d9eadd18917508894cf266b4c7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.976974 32357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "27e3d0d318a7440baa5f70966cd4236c"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "2b5ad78896be9ec4de2b86a42dd4df06"
server_key_iv: "31cdd8d9eadd18917508894cf266b4c7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.978181 32357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:38.978894 32371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.979046 32357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.979130 32357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "27e3d0d318a7440baa5f70966cd4236c"
format_stamp: "Formatted at 2026-09-28 02:43:38 on dist-test-slave-b5p5"
server_key: "2b5ad78896be9ec4de2b86a42dd4df06"
server_key_iv: "31cdd8d9eadd18917508894cf266b4c7"
server_key_version: "encryptionkey@0"
I20260928 02:43:38.979180 32357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:38.989873 32357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:38.992057 32357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:38.992161 32357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:38.992313 32357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:38.992606 32357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:38.992640 32357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.992659 32357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:38.992705 32357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:38.998134 32357 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:35385
I20260928 02:43:38.998201 32484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:35385 every 8 connection(s)
I20260928 02:43:38.998450 32357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:43:39 dist-test-slave-b5p5 krb5kdc[32118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:39.006947 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:38.999199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:44327 (local address 127.27.109.190:45641)
0928 02:43:38.999248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:38.999251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:38.999463 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:43:38.999869 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:38.999871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:38.999880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:38.999931 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.000738 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.001342 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.002245 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.002421 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.002524 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.004443 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.004445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.004445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:39.004451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.005766 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.006123 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.006124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.006125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.006190 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.006403 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.006404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.006404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.006520 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.006576 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.006811 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.006885 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:39.007047 32487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:38.999182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45641 (local address 127.27.109.130:44327)
0928 02:43:38.999340 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:38.999371 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:38.999791 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:38.999978 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:38.999981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.000202 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:39.000667 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.000671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.001422 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.001422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.002173 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.002175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.002539 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.002541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.002635 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.003113 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:39.003120 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:39.004355 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:39.005837 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:39.005842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:39.005844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:39.006038 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:39.006260 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:39.006260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:39.006261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:39.006347 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:39.006570 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:39.006574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:39.006673 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.006753 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.006858 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260928 02:43:39.007511 32485 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45641
I20260928 02:43:39.007586 32485 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:39.007746 32485 heartbeater.cc:511] Master 127.27.109.190:45641 requested a full tablet report, sending...
I20260928 02:43:39.008124 32162 ts_manager.cc:194] Registered new tserver with Master: 27e3d0d318a7440baa5f70966cd4236c (127.27.109.130:35385)
I20260928 02:43:39.008469 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32357
I20260928 02:43:39.008498 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:44327
I20260928 02:43:39.008576 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:39.008839 28086 external_mini_cluster.cc:1428] Setting key 0170fda2bc94b4eef401ac8e07fef52c
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:39.032496 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:39.110803 32490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:39.111076 32490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:39.111117 32490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:39.112841 32490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:39.112900 32490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:39.112946 32490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:39.114408 32490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:45641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:39.114681 32490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:39.114943 32490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:39.119596 32496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.119617 32498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.119596 32495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.119760 32490 server_base.cc:1034] running on GCE node
I20260928 02:43:39.120011 32490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:39.120291 32490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:39.121480 32490 hybrid_clock.cc:648] HybridClock initialized: now 1790563419121478 us; error 47 us; skew 500 ppm
Sep 28 02:43:39 dist-test-slave-b5p5 krb5kdc[32118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563419, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:39.123961 32490 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:39.124312 32490 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:39.124383 32490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:39.124522 32490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:39.125532 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "0f5f32f869e47053d170041bc4ee5f4c"
server_key_iv: "f960c7ead7aa6bffd65e6468e5f906c4"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.125888 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "0f5f32f869e47053d170041bc4ee5f4c"
server_key_iv: "f960c7ead7aa6bffd65e6468e5f906c4"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.127218 32490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:39.127902 32504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.128067 32490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:39.128162 32490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "0f5f32f869e47053d170041bc4ee5f4c"
server_key_iv: "f960c7ead7aa6bffd65e6468e5f906c4"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.128240 32490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:39.144345 32490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:39.146191 32490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:39.146358 32490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:39.146652 32490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:39.146975 32490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:39.147029 32490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.147070 32490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:39.147099 32490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.152891 32490 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:37543
I20260928 02:43:39.152933 32617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:37543 every 8 connection(s)
I20260928 02:43:39.153196 32490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:39.156745 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32490
I20260928 02:43:39.156847 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:39.157043 28086 external_mini_cluster.cc:1428] Setting key 257518d243ce5a79fb5a2e31eec47566
Sep 28 02:43:39 dist-test-slave-b5p5 krb5kdc[32118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563419, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:39.162506 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.153856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:50615 (local address 127.27.109.190:45641)
0928 02:43:39.153901 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.153904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.154153 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:43:39.154665 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.154667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.154677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.154730 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.155603 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.156104 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.157404 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.157603 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.157689 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.159777 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.159779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.159779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:39.159787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.161164 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.161565 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.161566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.161567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.161622 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.161861 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.161862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.161862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.161968 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.162041 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.162304 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.162376 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:39.162662 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.153856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:45641 (local address 127.27.109.131:50615)
0928 02:43:39.154035 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.154069 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.154579 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.154802 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.154805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.155056 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:39.155526 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.155531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.156180 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.156180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.157313 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.157315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.157695 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.157696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.157822 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.158315 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:39.158323 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:39.159684 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:39.161261 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:39.161267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:39.161269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:39.161479 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:39.161688 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:39.161689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:39.161689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:39.161794 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:39.162044 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:39.162050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:39.162178 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.162281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.162423 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260928 02:43:39.163064 32618 heartbeater.cc:347] Connected to a master server at 127.27.109.190:45641
I20260928 02:43:39.163132 32618 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:39.163287 32618 heartbeater.cc:511] Master 127.27.109.190:45641 requested a full tablet report, sending...
I20260928 02:43:39.163635 32162 ts_manager.cc:194] Registered new tserver with Master: 3dc8f45f09bc4e6d94ede490fa0a5bba (127.27.109.131:37543)
I20260928 02:43:39.164001 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:50615
I20260928 02:43:39.168654 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:39.175414 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.169912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.27.109.190:45641)
0928 02:43:39.169973 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.169975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.170063 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:43:39.170098 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.170099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.170106 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.170154 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.170729 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.171164 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.171863 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.172047 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.172106 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.173123 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.173124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.173124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:39.173130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.174412 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.174758 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.174759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.174759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.174805 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:39.175014 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:39.175015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:39.175015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:39.175132 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.175203 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.175302 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.175351 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:39.176769 32162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:39.178243 32162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:39.184590 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.27.109.131:37543)
0928 02:43:39.180872 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.180876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.180887 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:39.180979 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.180981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.180997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.181067 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:39.181278 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182054 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.183740 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184195 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184300 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184353 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.184475 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.184511 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":79,"threads_started":1}
I20260928 02:43:39.184716 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:45251 (local address 127.0.0.1:47324)
0928 02:43:39.180509 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.180698 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.180760 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.181017 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.181017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.181020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:39.181056 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.181057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182305 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.182306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.183460 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.183462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184435 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.184520 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184523 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.184567 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.184600 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":44,"thread_start_us":62,"threads_started":1}
I20260928 02:43:39.184778 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.27.109.130:35385)
0928 02:43:39.180625 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.180630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.180645 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:43:39.180705 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.180707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.180726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.180864 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:39.181033 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182189 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.183666 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184308 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184365 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184418 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.184642 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.184696 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260928 02:43:39.184777 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.27.109.129:45251)
0928 02:43:39.180466 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.180471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.180767 (+ 296us) server_negotiation.cc:407] Connection header received
0928 02:43:39.180842 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.180844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.180860 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.180934 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:39.181131 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182210 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.183545 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184272 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184348 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184399 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.184642 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.184695 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260928 02:43:39.184599 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37543 (local address 127.0.0.1:38838)
0928 02:43:39.180517 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.180544 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.180614 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.181147 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.181148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.181150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:39.181195 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.181197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182178 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.182179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.183684 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.183685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184299 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.184375 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184387 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.184456 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.184531 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":51,"thread_start_us":70,"threads_started":1}
I20260928 02:43:39.185030 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.180202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35385 (local address 127.0.0.1:56028)
0928 02:43:39.180503 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.180544 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.180613 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.180862 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.180865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.180867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:39.180943 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.180945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.182289 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.182289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.183583 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.183586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.184446 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.184447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.184505 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.184509 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.184561 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.184600 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":32,"thread_start_us":57,"threads_started":1}
I20260928 02:43:39.185420 32286 tablet_service.cc:1467] Processing CreateTablet for tablet a24d758db2dd424cae026c3412186876 (DEFAULT_TABLE table=test-table [id=cef2bd0871ec4278b34c87a58b5c590a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:39.185701 32286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24d758db2dd424cae026c3412186876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:39.185617 32419 tablet_service.cc:1467] Processing CreateTablet for tablet a24d758db2dd424cae026c3412186876 (DEFAULT_TABLE table=test-table [id=cef2bd0871ec4278b34c87a58b5c590a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:39.185855 32419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24d758db2dd424cae026c3412186876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:39.188071 32632 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Bootstrap starting.
I20260928 02:43:39.188535 32633 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Bootstrap starting.
I20260928 02:43:39.188719 32632 tablet_bootstrap.cc:654] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:39.188968 32552 tablet_service.cc:1467] Processing CreateTablet for tablet a24d758db2dd424cae026c3412186876 (DEFAULT_TABLE table=test-table [id=cef2bd0871ec4278b34c87a58b5c590a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:39.189095 32633 tablet_bootstrap.cc:654] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:39.189303 32552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24d758db2dd424cae026c3412186876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:39.189366 32632 log.cc:824] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:39.189682 32633 log.cc:824] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:39.190061 32632 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: No bootstrap required, opened a new log
I20260928 02:43:39.190130 32632 ts_tablet_manager.cc:1397] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:39.190508 32633 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: No bootstrap required, opened a new log
I20260928 02:43:39.190573 32633 ts_tablet_manager.cc:1397] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:39.192119 32633 raft_consensus.cc:348] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.192124 32632 raft_consensus.cc:348] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.192266 32632 raft_consensus.cc:374] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:39.192271 32633 raft_consensus.cc:374] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:39.192312 32633 raft_consensus.cc:729] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e3d0d318a7440baa5f70966cd4236c, State: Initialized, Role: FOLLOWER
I20260928 02:43:39.192308 32632 raft_consensus.cc:729] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e063afac3cd47bdb4acf452addf174d, State: Initialized, Role: FOLLOWER
I20260928 02:43:39.192422 32632 consensus_queue.cc:260] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.192485 32633 consensus_queue.cc:260] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.192678 32632 ts_tablet_manager.cc:1428] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:39.192749 32633 ts_tablet_manager.cc:1428] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:39.192742 32636 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Bootstrap starting.
I20260928 02:43:39.192806 32352 heartbeater.cc:503] Master 127.27.109.190:45641 was elected leader, sending a full tablet report...
I20260928 02:43:39.192865 32485 heartbeater.cc:503] Master 127.27.109.190:45641 was elected leader, sending a full tablet report...
I20260928 02:43:39.193331 32636 tablet_bootstrap.cc:654] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:39.193917 32636 log.cc:824] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:39.194617 32636 tablet_bootstrap.cc:492] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: No bootstrap required, opened a new log
I20260928 02:43:39.194666 32636 ts_tablet_manager.cc:1397] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:39.196039 32636 raft_consensus.cc:348] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.196146 32636 raft_consensus.cc:374] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:39.196166 32636 raft_consensus.cc:729] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc8f45f09bc4e6d94ede490fa0a5bba, State: Initialized, Role: FOLLOWER
I20260928 02:43:39.196234 32636 consensus_queue.cc:260] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.196412 32636 ts_tablet_manager.cc:1428] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:43:39.196470 32618 heartbeater.cc:503] Master 127.27.109.190:45641 was elected leader, sending a full tablet report...
W20260928 02:43:39.248992 32486 tablet.cc:2369] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:39.353644 32353 tablet.cc:2369] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:43:39.403703 32619 tablet.cc:2369] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:43:39.488036 32637 raft_consensus.cc:482] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:39.488210 32637 raft_consensus.cc:504] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.488667 32637 leader_election.cc:288] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27e3d0d318a7440baa5f70966cd4236c (127.27.109.130:35385), 3dc8f45f09bc4e6d94ede490fa0a5bba (127.27.109.131:37543)
I20260928 02:43:39.491804 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.488940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:44343 (local address 127.27.109.131:37543)
0928 02:43:39.488991 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.488993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.489102 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:43:39.489211 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.489213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.489222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.489296 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:39.489429 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.490027 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491067 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.491466 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491523 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.491530 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.491681 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.491719 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:43:39.491797 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.488861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37543 (local address 127.27.109.129:44343)
0928 02:43:39.489028 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.489042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.489117 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.489300 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.489300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.489303 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:39.489357 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.489359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.490100 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.490101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.490983 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.490985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.491543 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.491605 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.491609 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.491662 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.491698 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":28,"thread_start_us":65,"threads_started":1}
I20260928 02:43:39.491909 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.489015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:49881 (local address 127.27.109.130:35385)
0928 02:43:39.489101 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.489103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.489117 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:43:39.489215 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.489216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.489224 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.489298 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:39.489500 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.490048 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491139 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.491528 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491598 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.491607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.491791 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.491839 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:43:39.491925 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.488932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35385 (local address 127.27.109.129:49881)
0928 02:43:39.489039 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.489050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.489134 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.489313 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.489314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.489316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:39.489390 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.489409 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.490131 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.490132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.491030 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:39.491033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.491618 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.491618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:39.491704 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.491708 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.491764 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.491799 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260928 02:43:39.492259 32439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24d758db2dd424cae026c3412186876" candidate_uuid: "0e063afac3cd47bdb4acf452addf174d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e3d0d318a7440baa5f70966cd4236c" is_pre_election: true
I20260928 02:43:39.492419 32439 raft_consensus.cc:2454] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e063afac3cd47bdb4acf452addf174d in term 0.
I20260928 02:43:39.492532 32572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24d758db2dd424cae026c3412186876" candidate_uuid: "0e063afac3cd47bdb4acf452addf174d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" is_pre_election: true
I20260928 02:43:39.492641 32239 leader_election.cc:302] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e063afac3cd47bdb4acf452addf174d, 27e3d0d318a7440baa5f70966cd4236c; no voters:
I20260928 02:43:39.492692 32572 raft_consensus.cc:2454] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e063afac3cd47bdb4acf452addf174d in term 0.
I20260928 02:43:39.492764 32637 raft_consensus.cc:2781] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:39.492830 32637 raft_consensus.cc:482] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:39.492899 32637 raft_consensus.cc:3037] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:39.493600 32637 raft_consensus.cc:504] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.493737 32637 leader_election.cc:288] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [CANDIDATE]: Term 1 election: Requested vote from peers 27e3d0d318a7440baa5f70966cd4236c (127.27.109.130:35385), 3dc8f45f09bc4e6d94ede490fa0a5bba (127.27.109.131:37543)
I20260928 02:43:39.493867 32439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24d758db2dd424cae026c3412186876" candidate_uuid: "0e063afac3cd47bdb4acf452addf174d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e3d0d318a7440baa5f70966cd4236c"
I20260928 02:43:39.493911 32572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a24d758db2dd424cae026c3412186876" candidate_uuid: "0e063afac3cd47bdb4acf452addf174d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba"
I20260928 02:43:39.493966 32439 raft_consensus.cc:3037] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:39.494004 32572 raft_consensus.cc:3037] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:39.494697 32572 raft_consensus.cc:2454] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e063afac3cd47bdb4acf452addf174d in term 1.
I20260928 02:43:39.494796 32439 raft_consensus.cc:2454] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e063afac3cd47bdb4acf452addf174d in term 1.
I20260928 02:43:39.494872 32240 leader_election.cc:302] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e063afac3cd47bdb4acf452addf174d, 3dc8f45f09bc4e6d94ede490fa0a5bba; no voters:
I20260928 02:43:39.494963 32637 raft_consensus.cc:2781] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:39.495095 32637 raft_consensus.cc:686] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 1 LEADER]: Becoming Leader. State: Replica: 0e063afac3cd47bdb4acf452addf174d, State: Running, Role: LEADER
I20260928 02:43:39.495317 32637 consensus_queue.cc:237] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } }
I20260928 02:43:39.496138 32161 catalog_manager.cc:5495] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e063afac3cd47bdb4acf452addf174d (127.27.109.129). New cstate: current_term: 1 leader_uuid: "0e063afac3cd47bdb4acf452addf174d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e063afac3cd47bdb4acf452addf174d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 } health_report { overall_health: UNKNOWN } } }
I20260928 02:43:39.534950 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.532636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.27.109.129:45251)
0928 02:43:39.532684 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.532685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.532692 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:39.532725 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.532725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.532742 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.532800 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:39.532927 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.533433 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.534195 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:39.534371 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:39.534425 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:39.534608 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:39.534752 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:39.534814 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.534865 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.534882 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:39.536761 32572 raft_consensus.cc:1261] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 1 FOLLOWER]: Refusing update from remote peer 0e063afac3cd47bdb4acf452addf174d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:39.536761 32439 raft_consensus.cc:1261] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 1 FOLLOWER]: Refusing update from remote peer 0e063afac3cd47bdb4acf452addf174d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:39.537137 32643 consensus_queue.cc:1035] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e3d0d318a7440baa5f70966cd4236c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:39.537336 32637 consensus_queue.cc:1035] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc8f45f09bc4e6d94ede490fa0a5bba" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:39.539513 32646 mvcc.cc:205] Tried to move back new op lower bound from 7334147766419890176 to 7334147766253424640. Current Snapshot: MvccSnapshot[applied={T|T < 7334147766419890176}]
I20260928 02:43:39.539861 32647 mvcc.cc:205] Tried to move back new op lower bound from 7334147766419890176 to 7334147766253424640. Current Snapshot: MvccSnapshot[applied={T|T < 7334147766419890176}]
I20260928 02:43:39.539917 32648 mvcc.cc:205] Tried to move back new op lower bound from 7334147766419890176 to 7334147766253424640. Current Snapshot: MvccSnapshot[applied={T|T < 7334147766419890176}]
I20260928 02:43:39.542209 32161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:39.542308 32161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:39.543092 32161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f608ac611c84c64b8b8dae945acb568: Sending DeleteTablet for 3 replicas of tablet a24d758db2dd424cae026c3412186876
I20260928 02:43:39.543453 32552 tablet_service.cc:1514] Processing DeleteTablet for tablet a24d758db2dd424cae026c3412186876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:38838
I20260928 02:43:39.543447 32286 tablet_service.cc:1514] Processing DeleteTablet for tablet a24d758db2dd424cae026c3412186876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:47324
I20260928 02:43:39.543493 32419 tablet_service.cc:1514] Processing DeleteTablet for tablet a24d758db2dd424cae026c3412186876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:56028
I20260928 02:43:39.543679 32655 tablet_replica.cc:330] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d: stopping tablet replica
I20260928 02:43:39.543812 32655 raft_consensus.cc:2229] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:39.543912 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32224
I20260928 02:43:39.544014 32655 raft_consensus.cc:2258] T a24d758db2dd424cae026c3412186876 P 0e063afac3cd47bdb4acf452addf174d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:39.545358 32657 tablet_replica.cc:330] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: stopping tablet replica
I20260928 02:43:39.545472 32657 raft_consensus.cc:2229] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:39.545570 32657 raft_consensus.cc:2258] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:39.545728 32656 tablet_replica.cc:330] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: stopping tablet replica
I20260928 02:43:39.545817 32656 raft_consensus.cc:2229] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:39.545914 32656 raft_consensus.cc:2258] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:39.546211 32657 ts_tablet_manager.cc:1905] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:39.546839 32656 ts_tablet_manager.cc:1905] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:39.548215 32657 ts_tablet_manager.cc:1918] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:39.548247 32656 ts_tablet_manager.cc:1918] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:39.548290 32657 log.cc:1196] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/a24d758db2dd424cae026c3412186876
I20260928 02:43:39.548331 32656 log.cc:1196] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/a24d758db2dd424cae026c3412186876
I20260928 02:43:39.548583 32656 ts_tablet_manager.cc:1939] T a24d758db2dd424cae026c3412186876 P 27e3d0d318a7440baa5f70966cd4236c: Deleting consensus metadata
I20260928 02:43:39.548586 32657 ts_tablet_manager.cc:1939] T a24d758db2dd424cae026c3412186876 P 3dc8f45f09bc4e6d94ede490fa0a5bba: Deleting consensus metadata
I20260928 02:43:39.548962 32146 catalog_manager.cc:4841] TS 27e3d0d318a7440baa5f70966cd4236c (127.27.109.130:35385): tablet a24d758db2dd424cae026c3412186876 (table test-table [id=cef2bd0871ec4278b34c87a58b5c590a]) successfully deleted
I20260928 02:43:39.549140 32147 catalog_manager.cc:4841] TS 3dc8f45f09bc4e6d94ede490fa0a5bba (127.27.109.131:37543): tablet a24d758db2dd424cae026c3412186876 (table test-table [id=cef2bd0871ec4278b34c87a58b5c590a]) successfully deleted
W20260928 02:43:39.549569 32147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:45251 (error 108)
I20260928 02:43:39.549759 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32357
I20260928 02:43:39.550307 32626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.549863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:45251 (local address 127.0.0.1:47344)
0928 02:43:39.550239 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.550273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260928 02:43:39.550611 32147 catalog_manager.cc:4584] TS 0e063afac3cd47bdb4acf452addf174d (127.27.109.129:45251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a24d758db2dd424cae026c3412186876: Network error: Client connection negotiation failed: client connection to 127.27.109.129:45251: connect: Connection refused (error 111)
I20260928 02:43:39.554232 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32490
I20260928 02:43:39.558713 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32131
2026-09-28T02:43:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2144 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:43:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:39Z Disabled control of system clock
I20260928 02:43:39.585655 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40575
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:43:39.663949 32662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:39.664134 32662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:39.664165 32662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:39.665315 32662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:39.665357 32662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:39.665370 32662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:39.665383 32662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:39.665433 32662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:39.666801 32662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:39.667034 32662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:39.667320 32662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:39.672089 32670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.672124 32662 server_base.cc:1034] running on GCE node
W20260928 02:43:39.672112 32667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.672089 32668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.672479 32662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:39.672699 32662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:39.673834 32662 hybrid_clock.cc:648] HybridClock initialized: now 1790563419673822 us; error 29 us; skew 500 ppm
I20260928 02:43:39.675071 32662 webserver.cc:460] Webserver started at http://127.27.109.190:32923/ using document root <none> and password file <none>
I20260928 02:43:39.675294 32662 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:39.675341 32662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:39.675490 32662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:39.676479 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "966b6d4460f74889be3a5f439af08c38"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "9605042f0a44da98ad7edcde4a01e965"
server_key_iv: "3f9e7a3d7b5182fa09bbdc568f44dc78"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.676819 32662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "966b6d4460f74889be3a5f439af08c38"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "9605042f0a44da98ad7edcde4a01e965"
server_key_iv: "3f9e7a3d7b5182fa09bbdc568f44dc78"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.678009 32662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:39.678805 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.678993 32662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.679090 32662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "966b6d4460f74889be3a5f439af08c38"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "9605042f0a44da98ad7edcde4a01e965"
server_key_iv: "3f9e7a3d7b5182fa09bbdc568f44dc78"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.679167 32662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:39.682183 32662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:39.682431 32662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:39.682595 32662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:39.686141 32662 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:40575
I20260928 02:43:39.686172 32728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:40575 every 8 connection(s)
I20260928 02:43:39.686514 32662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:39.687110 32729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:39.689359 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32662
I20260928 02:43:39.689446 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:39.689623 28086 external_mini_cluster.cc:1428] Setting key bc2f2e05206ef0b28754f6f4602bc34f
I20260928 02:43:39.689707 32729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: Bootstrap starting.
I20260928 02:43:39.690404 32729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:39.690775 32729 log.cc:824] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:39.691521 32729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: No bootstrap required, opened a new log
I20260928 02:43:39.692111 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.690061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.27.109.190:40575)
0928 02:43:39.690253 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.690261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.690292 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:43:39.690874 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.690882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.690936 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.691075 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.691714 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.691717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.691717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:39.691721 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.691771 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:39.691781 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.691874 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.691906 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.691961 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260928 02:43:39.693498 32729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } }
I20260928 02:43:39.693643 32729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:39.693691 32729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966b6d4460f74889be3a5f439af08c38, State: Initialized, Role: FOLLOWER
I20260928 02:43:39.693831 32729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } }
I20260928 02:43:39.693923 32729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:39.693967 32729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:39.694013 32729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:39.694553 32729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } }
I20260928 02:43:39.694662 32729 leader_election.cc:302] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966b6d4460f74889be3a5f439af08c38; no voters:
I20260928 02:43:39.694835 32729 leader_election.cc:288] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:39.694864 32734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:39.695021 32734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [term 1 LEADER]: Becoming Leader. State: Replica: 966b6d4460f74889be3a5f439af08c38, State: Running, Role: LEADER
I20260928 02:43:39.695079 32729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:39.695161 32734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } }
I20260928 02:43:39.695511 32736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966b6d4460f74889be3a5f439af08c38. Latest consensus state: current_term: 1 leader_uuid: "966b6d4460f74889be3a5f439af08c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } } }
I20260928 02:43:39.695489 32735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966b6d4460f74889be3a5f439af08c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966b6d4460f74889be3a5f439af08c38" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40575 } } }
I20260928 02:43:39.695587 32736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:39.695595 32735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:39.695792 32742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:39.696259 32742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:39.697657 32742 catalog_manager.cc:1347] Generated new cluster ID: 7fe84f0c6b79492fbae83e41aecbc33e
I20260928 02:43:39.697700 32742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:39.715406 32742 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:39.715560 32742 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:39.728749 32742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: Generated new TSK 0
I20260928 02:43:39.728926 32742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:39.734869 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--builtin_ntp_servers=127.27.109.148:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:43:39.827692 32753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:39.827876 32753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:39.827898 32753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:39.829003 32753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:39.829041 32753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:39.829066 32753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:39.830430 32753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:39.830714 32753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:39.830996 32753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:39.835659 32759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.835650 32761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.835822 32753 server_base.cc:1034] running on GCE node
W20260928 02:43:39.835918 32758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.836127 32753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:39.836361 32753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:39.837518 32753 hybrid_clock.cc:648] HybridClock initialized: now 1790563419837500 us; error 28 us; skew 500 ppm
I20260928 02:43:39.838804 32753 webserver.cc:460] Webserver started at http://127.27.109.129:44235/ using document root <none> and password file <none>
I20260928 02:43:39.839058 32753 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:39.839129 32753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:39.839279 32753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:39.840305 32753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "793a2e57c50645f6822ded529aa290cc"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "6c30e044b0fb59c276cb8f50d7b88b3b"
server_key_iv: "2883297029b0bc0fd275f816a5d77ce0"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.840641 32753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "793a2e57c50645f6822ded529aa290cc"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "6c30e044b0fb59c276cb8f50d7b88b3b"
server_key_iv: "2883297029b0bc0fd275f816a5d77ce0"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.841823 32753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:39.842450 32767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.842635 32753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.842727 32753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "793a2e57c50645f6822ded529aa290cc"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "6c30e044b0fb59c276cb8f50d7b88b3b"
server_key_iv: "2883297029b0bc0fd275f816a5d77ce0"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.842806 32753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:39.845844 32753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:39.846091 32753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:39.846237 32753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:39.846443 32753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:39.846766 32753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:39.846819 32753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.846858 32753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:39.846892 32753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.852357 32753 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:35411
I20260928 02:43:39.852424 412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:35411 every 8 connection(s)
I20260928 02:43:39.852713 32753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:39.855428 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.853422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33405 (local address 127.27.109.190:40575)
0928 02:43:39.853473 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.853475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.853700 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:43:39.854114 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.854115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.854120 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.854170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.855255 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.855256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.855257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:39.855260 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.855273 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:39.855277 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.855322 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.855367 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.855377 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:39.855608 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.853423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40575 (local address 127.27.109.129:33405)
0928 02:43:39.853589 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.853620 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.854033 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.854227 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.854231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.854238 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:39.855116 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:39.855117 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:39.855143 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:39.855168 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:39.855320 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:39.855322 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:39.855324 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.855372 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.855415 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260928 02:43:39.856055 413 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40575
I20260928 02:43:39.856127 413 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:39.856285 413 heartbeater.cc:511] Master 127.27.109.190:40575 requested a full tablet report, sending...
I20260928 02:43:39.856757 32693 ts_manager.cc:194] Registered new tserver with Master: 793a2e57c50645f6822ded529aa290cc (127.27.109.129:35411)
I20260928 02:43:39.859845 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 32753
I20260928 02:43:39.859927 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:39.860064 28086 external_mini_cluster.cc:1428] Setting key 461aca6e9ad173e85ce1a57afd92a111
I20260928 02:43:39.861033 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--builtin_ntp_servers=127.27.109.148:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:43:39.940052 416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:39.940240 416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:39.940263 416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:39.941473 416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:39.941535 416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:39.941579 416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:39.943240 416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:39.943423 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:39.943645 416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:39.948400 424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.948361 421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:39.948356 422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:39.948832 416 server_base.cc:1034] running on GCE node
I20260928 02:43:39.949059 416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:39.949287 416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:39.950428 416 hybrid_clock.cc:648] HybridClock initialized: now 1790563419950415 us; error 29 us; skew 500 ppm
I20260928 02:43:39.951680 416 webserver.cc:460] Webserver started at http://127.27.109.130:41817/ using document root <none> and password file <none>
I20260928 02:43:39.951921 416 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:39.951968 416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:39.952112 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:39.953114 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "1d9f4bf0be274fe79f588b27b5f868a8"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "491b6262230427c25a3bea3607b37bcb"
server_key_iv: "fdb7041bef4d8957e6f788f647f3763b"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.953451 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "1d9f4bf0be274fe79f588b27b5f868a8"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "491b6262230427c25a3bea3607b37bcb"
server_key_iv: "fdb7041bef4d8957e6f788f647f3763b"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.954704 416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:39.955404 430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.955554 416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.955648 416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "1d9f4bf0be274fe79f588b27b5f868a8"
format_stamp: "Formatted at 2026-09-28 02:43:39 on dist-test-slave-b5p5"
server_key: "491b6262230427c25a3bea3607b37bcb"
server_key_iv: "fdb7041bef4d8957e6f788f647f3763b"
server_key_version: "encryptionkey@0"
I20260928 02:43:39.955724 416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:39.958904 416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:39.959133 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:39.959268 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:39.959463 416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:39.959740 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:39.959792 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.959830 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:39.959857 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:39.965236 416 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:33371
I20260928 02:43:39.965312 543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:33371 every 8 connection(s)
I20260928 02:43:39.965619 416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:39.968330 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.966320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:55355 (local address 127.27.109.190:40575)
0928 02:43:39.966397 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:39.966400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:39.966703 (+ 303us) server_negotiation.cc:407] Connection header received
0928 02:43:39.967097 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:39.967097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:39.967103 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:39.967153 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:39.968157 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:39.968158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:39.968158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:39.968162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:39.968174 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:39.968178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:39.968221 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:39.968269 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:43:39.968281 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:39.968492 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:39.966303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40575 (local address 127.27.109.130:55355)
0928 02:43:39.966595 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:39.966625 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:39.967023 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:39.967200 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:39.967203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:39.967210 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:39.968028 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:39.968028 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:39.968053 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:39.968079 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:39.968220 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:39.968222 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:39.968224 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:39.968268 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:43:39.968315 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260928 02:43:39.968995 544 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40575
I20260928 02:43:39.969072 544 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:39.969229 544 heartbeater.cc:511] Master 127.27.109.190:40575 requested a full tablet report, sending...
I20260928 02:43:39.969565 32693 ts_manager.cc:194] Registered new tserver with Master: 1d9f4bf0be274fe79f588b27b5f868a8 (127.27.109.130:33371)
I20260928 02:43:39.975417 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 416
I20260928 02:43:39.975505 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:39.975623 28086 external_mini_cluster.cc:1428] Setting key 63314848092e0de87011c01c2d9951e1
I20260928 02:43:39.976471 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--builtin_ntp_servers=127.27.109.148:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:43:40.054895 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:40.055081 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:40.055101 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:40.056143 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:40.056173 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:40.056197 547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:40.057407 547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:40.057582 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:40.057802 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:40.062719 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.062737 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.062820 547 server_base.cc:1034] running on GCE node
W20260928 02:43:40.062726 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.063102 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:40.063329 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:40.064484 547 hybrid_clock.cc:648] HybridClock initialized: now 1790563420064475 us; error 42 us; skew 500 ppm
I20260928 02:43:40.065706 547 webserver.cc:460] Webserver started at http://127.27.109.131:43851/ using document root <none> and password file <none>
I20260928 02:43:40.065948 547 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:40.066018 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:40.066165 547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:40.067217 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "c5fd5594cff54856b163ca07c2ba5314"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "33c69ed908357e9524b8cd6166734f2f"
server_key_iv: "d974afd8566e9e883d183aa1da2a1827"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.067566 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "c5fd5594cff54856b163ca07c2ba5314"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "33c69ed908357e9524b8cd6166734f2f"
server_key_iv: "d974afd8566e9e883d183aa1da2a1827"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.068828 547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:40.069514 561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.069703 547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:40.069797 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "c5fd5594cff54856b163ca07c2ba5314"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "33c69ed908357e9524b8cd6166734f2f"
server_key_iv: "d974afd8566e9e883d183aa1da2a1827"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.069885 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:40.073230 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:40.073487 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:40.073618 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:40.073815 547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:40.074111 547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:40.074163 547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.074203 547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:40.074230 547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.079783 547 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:36849
I20260928 02:43:40.079836 674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:36849 every 8 connection(s)
I20260928 02:43:40.080153 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:40.080603 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 547
I20260928 02:43:40.080711 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:40.080911 28086 external_mini_cluster.cc:1428] Setting key 19ecb4f3221f54bf0e92e74b4c596505
I20260928 02:43:40.083307 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.080908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:56995 (local address 127.27.109.190:40575)
0928 02:43:40.080987 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.080990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.081341 (+ 351us) server_negotiation.cc:407] Connection header received
0928 02:43:40.081893 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.081894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.081898 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.081932 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.083111 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.083112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.083112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.083116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.083129 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.083133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.083188 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.083240 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.083250 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:43:40.083460 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.080884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40575 (local address 127.27.109.131:56995)
0928 02:43:40.081156 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.081195 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.081790 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.081993 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.081996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.082004 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.082986 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.082986 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.083011 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.083035 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.083189 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.083192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.083194 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.083238 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.083279 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260928 02:43:40.083880 675 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40575
I20260928 02:43:40.083956 675 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:40.084103 675 heartbeater.cc:511] Master 127.27.109.190:40575 requested a full tablet report, sending...
I20260928 02:43:40.084445 32693 ts_manager.cc:194] Registered new tserver with Master: c5fd5594cff54856b163ca07c2ba5314 (127.27.109.131:36849)
I20260928 02:43:40.092772 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:43:40.095276 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.094079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.27.109.190:40575)
0928 02:43:40.094134 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.094135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.094214 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:43:40.094272 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.094273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.094277 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.094317 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.094973 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.094974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.094974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.094978 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.094991 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.094994 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.095051 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.095175 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.095190 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:43:40.097004 32693 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:40.098534 32693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:40.102259 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.100637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:36849 (local address 127.0.0.1:50094)
0928 02:43:40.100964 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.100974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.101008 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.101290 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.101295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.101302 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.101860 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.101861 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.101879 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.101909 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.102103 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.102108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.102111 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.102181 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.102194 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":89,"threads_started":1}
I20260928 02:43:40.102284 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.100865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.27.109.131:36849)
0928 02:43:40.101126 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.101131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.101143 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:43:40.101169 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.101171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.101181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.101523 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.101986 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.101994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.101995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.101998 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.102032 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.102053 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.102111 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.102186 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.102202 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260928 02:43:40.102558 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.100631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33371 (local address 127.0.0.1:39922)
0928 02:43:40.100794 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.100832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.100885 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.101396 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.101397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.101399 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.102141 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.102142 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.102146 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.102155 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.102409 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.102409 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.102410 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.102455 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.102496 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260928 02:43:40.102751 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.100709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.27.109.130:33371)
0928 02:43:40.100888 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.100893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.100904 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:40.100996 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.101001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.101012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.101082 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.102244 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.102253 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.102254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.102258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.102298 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.102325 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.102383 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.102657 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.102676 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260928 02:43:40.102905 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.100642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:35411 (local address 127.0.0.1:57604)
0928 02:43:40.101083 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.101091 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.101121 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.101401 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.101402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.101404 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.101987 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.101988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.101996 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.102008 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.102761 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.102762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.102763 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.102791 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.102849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":83,"threads_started":1}
I20260928 02:43:40.103020 609 tablet_service.cc:1467] Processing CreateTablet for tablet 16cafecca90541ae975d3355964edf01 (DEFAULT_TABLE table=test-table [id=5c3c43a65a3c4838887e232af88e4319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:40.103163 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.101057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.27.109.129:35411)
0928 02:43:40.101248 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.101252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.101262 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:40.101286 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.101287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.101295 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.101369 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.102294 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.102297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.102298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.102620 (+ 322us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.102655 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.102674 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.103010 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.103072 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.103088 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260928 02:43:40.103375 609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cafecca90541ae975d3355964edf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.103823 347 tablet_service.cc:1467] Processing CreateTablet for tablet 16cafecca90541ae975d3355964edf01 (DEFAULT_TABLE table=test-table [id=5c3c43a65a3c4838887e232af88e4319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:40.104122 347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cafecca90541ae975d3355964edf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.106693 689 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Bootstrap starting.
I20260928 02:43:40.107131 478 tablet_service.cc:1467] Processing CreateTablet for tablet 16cafecca90541ae975d3355964edf01 (DEFAULT_TABLE table=test-table [id=5c3c43a65a3c4838887e232af88e4319]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:40.107400 478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16cafecca90541ae975d3355964edf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.107422 689 tablet_bootstrap.cc:654] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.107697 689 log.cc:824] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.108398 689 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: No bootstrap required, opened a new log
I20260928 02:43:40.108482 689 ts_tablet_manager.cc:1397] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:40.110109 691 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Bootstrap starting.
I20260928 02:43:40.110651 689 raft_consensus.cc:348] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.110750 691 tablet_bootstrap.cc:654] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.110795 689 raft_consensus.cc:374] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.110836 689 raft_consensus.cc:729] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fd5594cff54856b163ca07c2ba5314, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.110965 689 consensus_queue.cc:260] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.111034 692 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: Bootstrap starting.
I20260928 02:43:40.111069 691 log.cc:824] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.111217 689 ts_tablet_manager.cc:1428] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:40.111354 675 heartbeater.cc:503] Master 127.27.109.190:40575 was elected leader, sending a full tablet report...
I20260928 02:43:40.111645 692 tablet_bootstrap.cc:654] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.111887 691 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: No bootstrap required, opened a new log
I20260928 02:43:40.111955 691 ts_tablet_manager.cc:1397] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:40.111969 692 log.cc:824] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.114246 691 raft_consensus.cc:348] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.114440 691 raft_consensus.cc:374] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.114518 691 raft_consensus.cc:729] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9f4bf0be274fe79f588b27b5f868a8, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.114636 691 consensus_queue.cc:260] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.114886 691 ts_tablet_manager.cc:1428] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:40.115025 544 heartbeater.cc:503] Master 127.27.109.190:40575 was elected leader, sending a full tablet report...
I20260928 02:43:40.115586 692 tablet_bootstrap.cc:492] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: No bootstrap required, opened a new log
I20260928 02:43:40.115651 692 ts_tablet_manager.cc:1397] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:43:40.117887 692 raft_consensus.cc:348] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.118006 692 raft_consensus.cc:374] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.118026 692 raft_consensus.cc:729] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a2e57c50645f6822ded529aa290cc, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.118098 692 consensus_queue.cc:260] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.118284 692 ts_tablet_manager.cc:1428] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:40.118333 413 heartbeater.cc:503] Master 127.27.109.190:40575 was elected leader, sending a full tablet report...
I20260928 02:43:40.153661 697 raft_consensus.cc:482] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:40.153826 697 raft_consensus.cc:504] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.154191 697 leader_election.cc:288] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5fd5594cff54856b163ca07c2ba5314 (127.27.109.131:36849), 1d9f4bf0be274fe79f588b27b5f868a8 (127.27.109.130:33371)
I20260928 02:43:40.155593 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.154403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33371 (local address 127.27.109.129:43953)
0928 02:43:40.154494 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.154508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.154553 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.154753 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.154754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.154757 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.155246 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.155246 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.155253 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.155263 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.155441 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.155442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.155443 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.155497 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.155512 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:43:40.155720 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.154509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:43953 (local address 127.27.109.130:33371)
0928 02:43:40.154559 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.154561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.154570 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:40.154648 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.154649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.154655 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.154717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.155345 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.155347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.155347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.155351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.155366 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.155371 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.155446 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.155628 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.155643 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:43:40.155762 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.154550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:59655 (local address 127.27.109.131:36849)
0928 02:43:40.154603 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.154605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.154762 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:43:40.154793 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.154794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.154800 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.154865 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.155523 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.155524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.155525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.155529 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.155544 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.155548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.155601 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.155679 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.155696 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:43:40.155814 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.154404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:36849 (local address 127.27.109.129:59655)
0928 02:43:40.154677 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.154695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.154740 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.154874 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.154875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.154887 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:40.155418 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:43:40.155419 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:40.155432 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:43:40.155443 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:40.155603 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:40.155604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:40.155605 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:43:40.155674 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:43:40.155735 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":99,"thread_start_us":109,"threads_started":1}
I20260928 02:43:40.156194 629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cafecca90541ae975d3355964edf01" candidate_uuid: "793a2e57c50645f6822ded529aa290cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5fd5594cff54856b163ca07c2ba5314" is_pre_election: true
I20260928 02:43:40.156190 498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cafecca90541ae975d3355964edf01" candidate_uuid: "793a2e57c50645f6822ded529aa290cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" is_pre_election: true
I20260928 02:43:40.156343 629 raft_consensus.cc:2454] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a2e57c50645f6822ded529aa290cc in term 0.
I20260928 02:43:40.156343 498 raft_consensus.cc:2454] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a2e57c50645f6822ded529aa290cc in term 0.
I20260928 02:43:40.156558 302 leader_election.cc:302] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a2e57c50645f6822ded529aa290cc, c5fd5594cff54856b163ca07c2ba5314; no voters:
I20260928 02:43:40.156682 697 raft_consensus.cc:2781] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:40.156737 697 raft_consensus.cc:482] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:40.156771 697 raft_consensus.cc:3037] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.157476 697 raft_consensus.cc:504] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.157634 697 leader_election.cc:288] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [CANDIDATE]: Term 1 election: Requested vote from peers c5fd5594cff54856b163ca07c2ba5314 (127.27.109.131:36849), 1d9f4bf0be274fe79f588b27b5f868a8 (127.27.109.130:33371)
I20260928 02:43:40.157765 629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cafecca90541ae975d3355964edf01" candidate_uuid: "793a2e57c50645f6822ded529aa290cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5fd5594cff54856b163ca07c2ba5314"
I20260928 02:43:40.157797 498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16cafecca90541ae975d3355964edf01" candidate_uuid: "793a2e57c50645f6822ded529aa290cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9f4bf0be274fe79f588b27b5f868a8"
I20260928 02:43:40.157848 629 raft_consensus.cc:3037] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.157858 498 raft_consensus.cc:3037] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.158792 629 raft_consensus.cc:2454] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a2e57c50645f6822ded529aa290cc in term 1.
I20260928 02:43:40.158792 498 raft_consensus.cc:2454] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a2e57c50645f6822ded529aa290cc in term 1.
I20260928 02:43:40.158965 302 leader_election.cc:302] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a2e57c50645f6822ded529aa290cc, c5fd5594cff54856b163ca07c2ba5314; no voters:
I20260928 02:43:40.159067 697 raft_consensus.cc:2781] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:40.159232 697 raft_consensus.cc:686] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [term 1 LEADER]: Becoming Leader. State: Replica: 793a2e57c50645f6822ded529aa290cc, State: Running, Role: LEADER
I20260928 02:43:40.159358 697 consensus_queue.cc:237] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } }
I20260928 02:43:40.160115 32693 catalog_manager.cc:5495] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a2e57c50645f6822ded529aa290cc (127.27.109.129). New cstate: current_term: 1 leader_uuid: "793a2e57c50645f6822ded529aa290cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "793a2e57c50645f6822ded529aa290cc" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 35411 } health_report { overall_health: HEALTHY } } }
I20260928 02:43:40.165300 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.164389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.27.109.129:35411)
0928 02:43:40.164434 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.164436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.164445 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:40.164513 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.164514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.164524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.164575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.165097 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.165099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.165099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.165103 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.165116 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.165120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.165165 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.165225 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.165237 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:43:40.166891 498 raft_consensus.cc:1261] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 1 FOLLOWER]: Refusing update from remote peer 793a2e57c50645f6822ded529aa290cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:40.166893 629 raft_consensus.cc:1261] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 1 FOLLOWER]: Refusing update from remote peer 793a2e57c50645f6822ded529aa290cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:40.167196 697 consensus_queue.cc:1035] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9f4bf0be274fe79f588b27b5f868a8" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:40.167316 699 consensus_queue.cc:1035] T 16cafecca90541ae975d3355964edf01 P 793a2e57c50645f6822ded529aa290cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5fd5594cff54856b163ca07c2ba5314" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:40.170256 703 mvcc.cc:205] Tried to move back new op lower bound from 7334147769001172992 to 7334147768973393920. Current Snapshot: MvccSnapshot[applied={T|T < 7334147769001172992}]
I20260928 02:43:40.170394 702 mvcc.cc:205] Tried to move back new op lower bound from 7334147769001172992 to 7334147768973393920. Current Snapshot: MvccSnapshot[applied={T|T < 7334147769001172992 or (T in {7334147769001172992})}]
I20260928 02:43:40.172155 32692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:40.172254 32692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:40.172945 32692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 966b6d4460f74889be3a5f439af08c38: Sending DeleteTablet for 3 replicas of tablet 16cafecca90541ae975d3355964edf01
I20260928 02:43:40.173290 609 tablet_service.cc:1514] Processing DeleteTablet for tablet 16cafecca90541ae975d3355964edf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:40 UTC) from {username='slave'} at 127.0.0.1:50094
I20260928 02:43:40.173318 478 tablet_service.cc:1514] Processing DeleteTablet for tablet 16cafecca90541ae975d3355964edf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:40 UTC) from {username='slave'} at 127.0.0.1:39922
I20260928 02:43:40.173523 347 tablet_service.cc:1514] Processing DeleteTablet for tablet 16cafecca90541ae975d3355964edf01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:40 UTC) from {username='slave'} at 127.0.0.1:57604
I20260928 02:43:40.173571 712 tablet_replica.cc:330] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: stopping tablet replica
I20260928 02:43:40.173661 712 raft_consensus.cc:2229] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:40.173750 712 raft_consensus.cc:2258] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:40.173739 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32753
I20260928 02:43:40.173904 713 tablet_replica.cc:330] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: stopping tablet replica
I20260928 02:43:40.173980 713 raft_consensus.cc:2229] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:40.174547 713 raft_consensus.cc:2258] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:40.175024 713 ts_tablet_manager.cc:1905] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:40.175024 712 ts_tablet_manager.cc:1905] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:40.176756 713 ts_tablet_manager.cc:1918] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:40.176759 712 ts_tablet_manager.cc:1918] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:40.176831 713 log.cc:1196] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/16cafecca90541ae975d3355964edf01
I20260928 02:43:40.176934 712 log.cc:1196] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/16cafecca90541ae975d3355964edf01
I20260928 02:43:40.177103 713 ts_tablet_manager.cc:1939] T 16cafecca90541ae975d3355964edf01 P 1d9f4bf0be274fe79f588b27b5f868a8: Deleting consensus metadata
I20260928 02:43:40.177281 712 ts_tablet_manager.cc:1939] T 16cafecca90541ae975d3355964edf01 P c5fd5594cff54856b163ca07c2ba5314: Deleting consensus metadata
I20260928 02:43:40.177528 32678 catalog_manager.cc:4841] TS 1d9f4bf0be274fe79f588b27b5f868a8 (127.27.109.130:33371): tablet 16cafecca90541ae975d3355964edf01 (table test-table [id=5c3c43a65a3c4838887e232af88e4319]) successfully deleted
I20260928 02:43:40.177742 32679 catalog_manager.cc:4841] TS c5fd5594cff54856b163ca07c2ba5314 (127.27.109.131:36849): tablet 16cafecca90541ae975d3355964edf01 (table test-table [id=5c3c43a65a3c4838887e232af88e4319]) successfully deleted
W20260928 02:43:40.179597 32680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:35411 (error 108)
I20260928 02:43:40.179847 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 416
I20260928 02:43:40.180054 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.179932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:35411 (local address 127.0.0.1:57622)
0928 02:43:40.179985 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.180017 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:35411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 02:43:40.180153 32680 catalog_manager.cc:4584] TS 793a2e57c50645f6822ded529aa290cc (127.27.109.129:35411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16cafecca90541ae975d3355964edf01: Network error: Client connection negotiation failed: client connection to 127.27.109.129:35411: connect: Connection refused (error 111)
I20260928 02:43:40.184988 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 547
I20260928 02:43:40.189705 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 32662
2026-09-28T02:43:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (630 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:43:40.209255 28086 mini_oidc.cc:152] Starting JWKS server
I20260928 02:43:40.210230 28086 webserver.cc:460] Webserver started at https://127.0.0.1:41909/ using document root <none> and password file <none>
I20260928 02:43:40.210355 28086 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:43:40.210589 28086 webserver.cc:460] Webserver started at http://127.0.0.1:40117/ using document root <none> and password file <none>
2026-09-28T02:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:40Z Disabled control of system clock
I20260928 02:43:40.218006 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40833
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40833
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41909/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/testchainca.cert with env {}
W20260928 02:43:40.296409 721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:40.296594 721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:40.296614 721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:40.297678 721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:40.297713 721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:40.297725 721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:40.297737 721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:40.297749 721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:43:40.297761 721 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:43:40.297780 721 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41909/jwks.json
I20260928 02:43:40.299254 721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41909/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:40.299440 721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:40.299592 721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:40.304797 729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.304838 726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.304941 727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.305150 721 server_base.cc:1034] running on GCE node
I20260928 02:43:40.305375 721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:40.305650 721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:40.306835 721 hybrid_clock.cc:648] HybridClock initialized: now 1790563420306818 us; error 43 us; skew 500 ppm
I20260928 02:43:40.308022 721 webserver.cc:460] Webserver started at http://127.27.109.190:34087/ using document root <none> and password file <none>
I20260928 02:43:40.308275 721 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:40.308348 721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:40.308477 721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:40.309506 721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "2b5d522804ec4f8581958c08e3bdfd95"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a02789e2cf6742e8ce4c392058b9cc7f"
server_key_iv: "0bb9de5cc0e02ac0bbc632f44691aee5"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.309860 721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "2b5d522804ec4f8581958c08e3bdfd95"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a02789e2cf6742e8ce4c392058b9cc7f"
server_key_iv: "0bb9de5cc0e02ac0bbc632f44691aee5"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.311232 721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:40.312063 735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.312232 721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:40.312336 721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "2b5d522804ec4f8581958c08e3bdfd95"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a02789e2cf6742e8ce4c392058b9cc7f"
server_key_iv: "0bb9de5cc0e02ac0bbc632f44691aee5"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.312417 721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:40.326143 721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:40.326409 721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:40.326591 721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:40.330116 721 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:40833
I20260928 02:43:40.330143 787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:40833 every 8 connection(s)
I20260928 02:43:40.330446 721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:40.331308 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.332206 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 721
I20260928 02:43:40.332320 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:40.332510 28086 external_mini_cluster.cc:1428] Setting key 8a0da3c8e54d68c2e466130a7293e655
I20260928 02:43:40.333822 788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95: Bootstrap starting.
I20260928 02:43:40.334383 788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.334702 788 log.cc:824] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.335326 788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95: No bootstrap required, opened a new log
I20260928 02:43:40.337095 788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } }
I20260928 02:43:40.337241 788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.337289 788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5d522804ec4f8581958c08e3bdfd95, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.337438 788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } }
I20260928 02:43:40.337530 788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:40.337568 788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:40.337628 788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.337800 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.332960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.27.109.190:40833)
0928 02:43:40.333153 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.333161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.333190 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:43:40.333659 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.333665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.333721 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.333873 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.334052 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.335019 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.335848 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.336235 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.336372 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.337054 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.337058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.337062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.337213 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.337242 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.337251 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.337324 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.337469 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.337551 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260928 02:43:40.338224 788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } }
I20260928 02:43:40.338333 788 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5d522804ec4f8581958c08e3bdfd95; no voters:
I20260928 02:43:40.338519 788 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:40.338549 793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:40.338652 793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [term 1 LEADER]: Becoming Leader. State: Replica: 2b5d522804ec4f8581958c08e3bdfd95, State: Running, Role: LEADER
I20260928 02:43:40.338640 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:40.338725 793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } }
I20260928 02:43:40.338771 788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:40.339021 794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5d522804ec4f8581958c08e3bdfd95. Latest consensus state: current_term: 1 leader_uuid: "2b5d522804ec4f8581958c08e3bdfd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } } }
I20260928 02:43:40.339035 795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5d522804ec4f8581958c08e3bdfd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d522804ec4f8581958c08e3bdfd95" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40833 } } }
I20260928 02:43:40.339111 794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.339123 795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.339308 798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:40.339780 798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:40.341183 798 catalog_manager.cc:1347] Generated new cluster ID: d5dfe906871b4579ad5faf1eaf654308
I20260928 02:43:40.341238 798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:40.349062 798 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:40.349540 798 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:40.361344 798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5d522804ec4f8581958c08e3bdfd95: Generated new TSK 0
I20260928 02:43:40.361512 798 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:43:40.381928 818 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:43:40.382025 818 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 02:43:40.382633 817 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:40.366932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40833 (local address 127.0.0.1:59368)
0928 02:43:40.367127 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.367135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.367187 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.367355 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.367356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.367358 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:43:40.367430 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.367432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.368352 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.368353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.368926 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.368927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.369419 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.369420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.369462 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.369477 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:43:40.382186 (+ 12709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:43:40.382280 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.190:40833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 02:43:40.382921 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.367010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59368 (local address 127.27.109.190:40833)
0928 02:43:40.367062 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.367064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.367185 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:43:40.367251 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.367252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.367265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.367351 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:43:40.367491 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.368268 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.368997 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.369329 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.369400 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.369568 (+ 168us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:43:40.382045 (+ 12477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:43:40.382844 (+ 799us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59368: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:43:40.382977 791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59368: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.109.190:40833: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41909/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:43:40.383329 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 721
2026-09-28T02:43:40Z chronyd exiting
I20260928 02:43:40.419013 28086 test_util.cc:175] -----------------------------------------------
I20260928 02:43:40.419080 28086 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563399140758-28086-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:43:40.419845 28086 mini_oidc.cc:152] Starting JWKS server
I20260928 02:43:40.420516 28086 webserver.cc:460] Webserver started at https://127.0.0.1:40731/ using document root <none> and password file <none>
I20260928 02:43:40.420555 28086 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:43:40.420733 28086 webserver.cc:460] Webserver started at http://127.0.0.1:34565/ using document root <none> and password file <none>
2026-09-28T02:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:40Z Disabled control of system clock
I20260928 02:43:40.427883 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46255
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:46255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/testchainca.cert with env {}
W20260928 02:43:40.505789 824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:40.506011 824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:40.506045 824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:40.507167 824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:40.507228 824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:40.507249 824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:40.507268 824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:40.507300 824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:43:40.507339 824 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:43:40.507364 824 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40731/jwks.json
I20260928 02:43:40.509032 824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:46255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:40.509313 824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:40.509523 824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:40.514429 829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.514498 832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.514454 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.514712 824 server_base.cc:1034] running on GCE node
I20260928 02:43:40.514930 824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:40.515151 824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:40.516314 824 hybrid_clock.cc:648] HybridClock initialized: now 1790563420516300 us; error 25 us; skew 500 ppm
I20260928 02:43:40.517472 824 webserver.cc:460] Webserver started at http://127.27.109.190:34619/ using document root <none> and password file <none>
I20260928 02:43:40.517699 824 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:40.517764 824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:40.517892 824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:40.518961 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a012be266d8dd35039605bef0f14616c"
server_key_iv: "1ae1d2a9587bc5074efe644506e783fc"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.519309 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a012be266d8dd35039605bef0f14616c"
server_key_iv: "1ae1d2a9587bc5074efe644506e783fc"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.520563 824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:40.521237 838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.521448 824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:40.521540 824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "a012be266d8dd35039605bef0f14616c"
server_key_iv: "1ae1d2a9587bc5074efe644506e783fc"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.521625 824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:40.540524 824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:40.540817 824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:40.540970 824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:40.544925 824 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:46255
I20260928 02:43:40.544957 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:46255 every 8 connection(s)
I20260928 02:43:40.545276 824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:40.545948 891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.548544 891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce: Bootstrap starting.
I20260928 02:43:40.549129 891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.549407 891 log.cc:824] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.550053 891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce: No bootstrap required, opened a new log
I20260928 02:43:40.551714 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 824
I20260928 02:43:40.551683 891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } }
I20260928 02:43:40.551801 891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.551829 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:40.551875 891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3354bb8caf0c4b1a9273f0f4698ea5ce, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.552016 891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } }
I20260928 02:43:40.552075 28086 external_mini_cluster.cc:1428] Setting key 8a38940c47a7f97a134a71c5253e4b46
I20260928 02:43:40.552088 891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:40.552116 891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:40.552156 891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.552757 891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } }
I20260928 02:43:40.552847 891 leader_election.cc:302] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3354bb8caf0c4b1a9273f0f4698ea5ce; no voters:
I20260928 02:43:40.553001 891 leader_election.cc:288] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:40.553047 896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:40.553191 896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [term 1 LEADER]: Becoming Leader. State: Replica: 3354bb8caf0c4b1a9273f0f4698ea5ce, State: Running, Role: LEADER
I20260928 02:43:40.553267 891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:40.553289 896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } }
I20260928 02:43:40.553745 897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } } }
I20260928 02:43:40.553838 897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.554026 898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3354bb8caf0c4b1a9273f0f4698ea5ce. Latest consensus state: current_term: 1 leader_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3354bb8caf0c4b1a9273f0f4698ea5ce" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46255 } } }
I20260928 02:43:40.554134 898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.554298 905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:40.554832 905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:40.556313 905 catalog_manager.cc:1347] Generated new cluster ID: e0b273eee36a4fb3b72a1ed98cc275e6
I20260928 02:43:40.556365 905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:40.558216 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.552526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.27.109.190:46255)
0928 02:43:40.552741 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.552749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.552851 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:43:40.552983 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.553008 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.553104 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.553508 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.553604 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.555241 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.556295 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.556752 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.556890 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.557568 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.557571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.557572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.557748 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.557780 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.557790 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.557875 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.557946 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.558022 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260928 02:43:40.559031 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:40.571897 905 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:40.572386 905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:40.575394 905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3354bb8caf0c4b1a9273f0f4698ea5ce: Generated new TSK 0
I20260928 02:43:40.575560 905 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:43:40.596120 921 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:43:40.596201 921 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:43:40.596486 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.582304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.27.109.190:46255)
0928 02:43:40.582347 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.582349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.582441 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:43:40.582549 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.582550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.582562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.582642 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:43:40.582756 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.583545 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.584275 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.584612 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.584678 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.584855 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:43:40.596283 (+ 11428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:43:40.596419 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 02:43:40.596537 895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:43:40.596519 920 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:40.582230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46255 (local address 127.0.0.1:60842)
0928 02:43:40.582365 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.582375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.582440 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.582645 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.582646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.582648 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:43:40.582701 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.582703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.583623 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.583624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.584195 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.584196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.584706 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.584706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.584756 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.584761 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:43:40.596385 (+ 11624us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:43:40.596458 (+ 73us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.190:46255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":91,"threads_started":1}
I20260928 02:43:40.596861 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 824
2026-09-28T02:43:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:43:40.633631 28086 mini_oidc.cc:152] Starting JWKS server
I20260928 02:43:40.634250 28086 webserver.cc:460] Webserver started at https://127.0.0.1:40163/ using document root <none> and password file <none>
I20260928 02:43:40.634289 28086 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:43:40.634423 28086 webserver.cc:460] Webserver started at http://127.0.0.1:34667/ using document root <none> and password file <none>
2026-09-28T02:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:40Z Disabled control of system clock
I20260928 02:43:40.641634 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39697
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39697
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40163/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:43:40.718446 927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:40.718640 927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:40.718660 927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:40.719754 927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:40.719790 927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:40.719802 927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:40.719815 927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:40.719827 927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:43:40.719839 927 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:43:40.719864 927 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40163/jwks.json
I20260928 02:43:40.721097 927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40163/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:40.721267 927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:40.721472 927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:40.726158 932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.726143 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.726174 935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.726221 927 server_base.cc:1034] running on GCE node
I20260928 02:43:40.726660 927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:40.726933 927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:40.728078 927 hybrid_clock.cc:648] HybridClock initialized: now 1790563420728070 us; error 38 us; skew 500 ppm
I20260928 02:43:40.729198 927 webserver.cc:460] Webserver started at http://127.27.109.190:44277/ using document root <none> and password file <none>
I20260928 02:43:40.729449 927 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:40.729519 927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:40.729664 927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:40.730726 927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "7960ebbb1d634917b85afbecfd7b4235"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "d5a591845eb326859eb1c55edfa5d081"
server_key_iv: "10cbaf457f61afc9069e89a30b6f3528"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.731071 927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "7960ebbb1d634917b85afbecfd7b4235"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "d5a591845eb326859eb1c55edfa5d081"
server_key_iv: "10cbaf457f61afc9069e89a30b6f3528"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.732244 927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:40.732889 941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.733047 927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.733134 927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "7960ebbb1d634917b85afbecfd7b4235"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "d5a591845eb326859eb1c55edfa5d081"
server_key_iv: "10cbaf457f61afc9069e89a30b6f3528"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.733211 927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:40.758837 927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:40.759166 927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:40.759332 927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:40.763193 927 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39697
I20260928 02:43:40.763230 993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39697 every 8 connection(s)
I20260928 02:43:40.763603 927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:40.764256 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.765337 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 927
I20260928 02:43:40.765470 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:40.765650 28086 external_mini_cluster.cc:1428] Setting key ff8fbbae74990cafb49bef74f58ffaab
I20260928 02:43:40.767829 994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235: Bootstrap starting.
I20260928 02:43:40.768497 994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.768786 994 log.cc:824] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.769428 994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235: No bootstrap required, opened a new log
I20260928 02:43:40.771380 994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } }
I20260928 02:43:40.771538 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.766152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.27.109.190:39697)
0928 02:43:40.766362 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.766370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.766403 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:40.766998 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.767006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.767090 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.767251 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.767428 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.768759 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.769552 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.769985 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.770119 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.770851 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.770855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.770856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.771017 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.771048 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.771058 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.771135 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.771225 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.771303 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 02:43:40.771545 994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.771649 994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7960ebbb1d634917b85afbecfd7b4235, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.771763 994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } }
I20260928 02:43:40.771855 994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:40.771895 994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:40.771942 994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.772464 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:40.772648 994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } }
I20260928 02:43:40.772764 994 leader_election.cc:302] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7960ebbb1d634917b85afbecfd7b4235; no voters:
I20260928 02:43:40.772938 994 leader_election.cc:288] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:40.772984 999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:40.773124 999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [term 1 LEADER]: Becoming Leader. State: Replica: 7960ebbb1d634917b85afbecfd7b4235, State: Running, Role: LEADER
I20260928 02:43:40.773182 994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:40.773217 999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } }
I20260928 02:43:40.773519 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7960ebbb1d634917b85afbecfd7b4235. Latest consensus state: current_term: 1 leader_uuid: "7960ebbb1d634917b85afbecfd7b4235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } } }
I20260928 02:43:40.773613 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.773617 1000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7960ebbb1d634917b85afbecfd7b4235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7960ebbb1d634917b85afbecfd7b4235" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39697 } } }
I20260928 02:43:40.773758 1000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.773878 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:40.774330 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:40.775905 1006 catalog_manager.cc:1347] Generated new cluster ID: 19294c30c49b4783802555efeadbbaad
I20260928 02:43:40.775957 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:40.800513 1006 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:40.801033 1006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:40.804796 1006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7960ebbb1d634917b85afbecfd7b4235: Generated new TSK 0
I20260928 02:43:40.804965 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:40.814342 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.810873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.27.109.190:39697)
0928 02:43:40.810930 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.810932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.811024 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:43:40.811099 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.811101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.811114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.811183 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:43:40.811276 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.812046 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.812768 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.813073 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.813139 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.813282 (+ 143us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:43:40.814154 (+ 872us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:43:40.814279 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 02:43:40.814404 997 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:43:40.814560 1023 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:40.810794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39697 (local address 127.0.0.1:57906)
0928 02:43:40.810955 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:40.810965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:40.811034 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:40.811187 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:40.811187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:40.811189 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:43:40.811234 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.811235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.812117 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.812117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.812694 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:40.812696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.813147 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.813148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:40.813192 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.813195 (+ 3us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:43:40.814249 (+ 1054us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:43:40.814336 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.190:39697: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40163/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260928 02:43:40.814844 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 927
E20260928 02:43:40.828599 1024 webserver.cc:533] Webserver: sslize failed:
E20260928 02:43:40.828657 1024 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:43:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:40Z Disabled control of system clock
I20260928 02:43:40.859215 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40937
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40937
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/testchainca.cert with env {}
W20260928 02:43:40.936646 1028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:40.936838 1028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:40.936859 1028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:40.938011 1028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:40.938051 1028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:40.938063 1028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:40.938076 1028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:40.938135 1028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:43:40.938158 1028 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:43:40.938177 1028 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:43:40.939666 1028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:40.939913 1028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:40.940122 1028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:40.944779 1036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.944792 1033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:40.944778 1034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:40.945040 1028 server_base.cc:1034] running on GCE node
I20260928 02:43:40.945276 1028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:40.945544 1028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:40.946718 1028 hybrid_clock.cc:648] HybridClock initialized: now 1790563420946704 us; error 24 us; skew 500 ppm
I20260928 02:43:40.947892 1028 webserver.cc:460] Webserver started at http://127.27.109.190:40489/ using document root <none> and password file <none>
I20260928 02:43:40.948132 1028 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:40.948199 1028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:40.948340 1028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:40.949334 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "bffec01403294d8597693b3c709f27a6"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "4e543d4f56462ec7a9210b08aae24767"
server_key_iv: "29a065b575aaa506ea5b6d53de9e1ae6"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.949688 1028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "bffec01403294d8597693b3c709f27a6"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "4e543d4f56462ec7a9210b08aae24767"
server_key_iv: "29a065b575aaa506ea5b6d53de9e1ae6"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.951002 1028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:40.951711 1042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:40.951910 1028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:40.952006 1028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "bffec01403294d8597693b3c709f27a6"
format_stamp: "Formatted at 2026-09-28 02:43:40 on dist-test-slave-b5p5"
server_key: "4e543d4f56462ec7a9210b08aae24767"
server_key_iv: "29a065b575aaa506ea5b6d53de9e1ae6"
server_key_version: "encryptionkey@0"
I20260928 02:43:40.952087 1028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:40.965554 1028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:40.965818 1028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:40.965960 1028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:40.969699 1028 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:40937
I20260928 02:43:40.969727 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:40937 every 8 connection(s)
I20260928 02:43:40.970038 1028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:40.970912 1095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:40.972851 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1028
I20260928 02:43:40.972963 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:40.973048 1095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6: Bootstrap starting.
I20260928 02:43:40.973152 28086 external_mini_cluster.cc:1428] Setting key 647e17657c6c04ed830b212280c86d4d
I20260928 02:43:40.973721 1095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:40.974006 1095 log.cc:824] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:40.974702 1095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6: No bootstrap required, opened a new log
I20260928 02:43:40.976763 1095 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } }
I20260928 02:43:40.976949 1095 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:40.976976 1095 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffec01403294d8597693b3c709f27a6, State: Initialized, Role: FOLLOWER
I20260928 02:43:40.977109 1095 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } }
I20260928 02:43:40.977195 1095 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:40.977216 1095 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:40.977283 1095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:40.977954 1095 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } }
I20260928 02:43:40.978060 1095 leader_election.cc:302] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffec01403294d8597693b3c709f27a6; no voters:
I20260928 02:43:40.978200 1095 leader_election.cc:288] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:40.978222 1100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:40.978322 1100 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [term 1 LEADER]: Becoming Leader. State: Replica: bffec01403294d8597693b3c709f27a6, State: Running, Role: LEADER
I20260928 02:43:40.978396 1100 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } }
I20260928 02:43:40.978430 1095 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:40.978958 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:40.973601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.27.109.190:40937)
0928 02:43:40.973809 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:40.973819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:40.973848 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:43:40.974301 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:40.974308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:40.974362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:40.974517 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:40.974689 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.975567 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.976753 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:40.977303 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:40.977438 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:40.978050 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:40.978055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:40.978056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:43:40.978271 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:40.978312 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:43:40.978328 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:40.978661 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:40.978704 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:43:40.978777 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 02:43:40.979305 1101 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffec01403294d8597693b3c709f27a6. Latest consensus state: current_term: 1 leader_uuid: "bffec01403294d8597693b3c709f27a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } } }
I20260928 02:43:40.979388 1101 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.979717 1102 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffec01403294d8597693b3c709f27a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec01403294d8597693b3c709f27a6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40937 } } }
I20260928 02:43:40.980079 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:43:40.979789 1102 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:40.979796 1111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:40.980500 1111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:40.981910 1111 catalog_manager.cc:1347] Generated new cluster ID: 9c494e3c7fc743d096009abb884f533f
I20260928 02:43:40.981967 1111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:43:40.992839 1111 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:40.993494 1111 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:40.998126 1111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffec01403294d8597693b3c709f27a6: Generated new TSK 0
I20260928 02:43:40.998275 1111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:43:41.882421 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:41.003700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.27.109.190:40937)
0928 02:43:41.003757 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:41.003759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:41.003863 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:43:41.003931 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:41.003932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:41.003943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:41.003992 (+ 49us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:43:41.004119 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:41.004577 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:41.005708 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:41.005889 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:41.005939 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:41.006108 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:43:41.882193 (+876085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:43:41.882350 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60452: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 02:43:41.882543 1098 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60452: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:43:41.882741 1124 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:41.003650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40937 (local address 127.0.0.1:60452)
0928 02:43:41.003794 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:41.003805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:41.003869 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:41.004001 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:41.004001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:41.004003 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:43:41.004065 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:41.004068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:41.004666 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:41.004667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:41.005621 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:41.005623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:41.005960 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:41.005961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:41.006020 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:41.006025 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:43:41.882344 (+876319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:43:41.882528 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.190:40937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260928 02:43:41.883149 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1028
2026-09-28T02:43:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1048 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:42 dist-test-slave-b5p5 krb5kdc[1136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:42 dist-test-slave-b5p5 krb5kdc[1136](info): set up 2 sockets
Sep 28 02:43:42 dist-test-slave-b5p5 krb5kdc[1136](info): commencing operation
krb5kdc: starting...
W20260928 02:43:43.087028 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:43:43.198992 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39433
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:43.277559 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:43.277783 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:43.277822 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:43.279030 1149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:43.279088 1149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:43.279110 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:43.279131 1149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:43.279160 1149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:43.280541 1149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:43.280802 1149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:43.281028 1149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:43.285653 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.285722 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.285652 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:43.285888 1149 server_base.cc:1034] running on GCE node
I20260928 02:43:43.286151 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:43.286427 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:43.287612 1149 hybrid_clock.cc:648] HybridClock initialized: now 1790563423287593 us; error 37 us; skew 500 ppm
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.290112 1149 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:43:43.290613 1149 webserver.cc:460] Webserver started at http://127.27.109.190:41969/ using document root <none> and password file <none>
I20260928 02:43:43.290838 1149 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:43.290917 1149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:43.291049 1149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:43.292044 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "e636fd848eee449b830a70ea3145c1bc"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d2c8ec4f142ffb9f1bf208f6f6920094"
server_key_iv: "6c348fba567c3800c92d4896cf222cdc"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.292408 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "e636fd848eee449b830a70ea3145c1bc"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d2c8ec4f142ffb9f1bf208f6f6920094"
server_key_iv: "6c348fba567c3800c92d4896cf222cdc"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.293671 1149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:43.294459 1164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.294693 1149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:43.294790 1149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "e636fd848eee449b830a70ea3145c1bc"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d2c8ec4f142ffb9f1bf208f6f6920094"
server_key_iv: "6c348fba567c3800c92d4896cf222cdc"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.294900 1149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:43.302627 1149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:43.304818 1149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:43.304998 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:43.308864 1149 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39433
I20260928 02:43:43.308887 1216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39433 every 8 connection(s)
I20260928 02:43:43.309203 1149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:43.309995 1217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:43.312096 1217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc: Bootstrap starting.
I20260928 02:43:43.312595 1217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:43.312819 1217 log.cc:824] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:43.313069 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1149
I20260928 02:43:43.313162 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:43.313354 28086 external_mini_cluster.cc:1428] Setting key f8e2c6653e05d1b531d822dcdcb82abe
I20260928 02:43:43.313530 1217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc: No bootstrap required, opened a new log
I20260928 02:43:43.315375 1217 raft_consensus.cc:348] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } }
I20260928 02:43:43.315526 1217 raft_consensus.cc:374] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:43.315590 1217 raft_consensus.cc:729] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e636fd848eee449b830a70ea3145c1bc, State: Initialized, Role: FOLLOWER
I20260928 02:43:43.315725 1217 consensus_queue.cc:260] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } }
I20260928 02:43:43.315824 1217 raft_consensus.cc:388] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:43.315876 1217 raft_consensus.cc:482] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:43.315927 1217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:43.316740 1217 raft_consensus.cc:504] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } }
I20260928 02:43:43.316865 1217 leader_election.cc:302] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e636fd848eee449b830a70ea3145c1bc; no voters:
I20260928 02:43:43.317046 1217 leader_election.cc:288] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:43.317085 1222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:43.317243 1222 raft_consensus.cc:686] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [term 1 LEADER]: Becoming Leader. State: Replica: e636fd848eee449b830a70ea3145c1bc, State: Running, Role: LEADER
I20260928 02:43:43.317286 1217 sys_catalog.cc:564] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:43.317363 1222 consensus_queue.cc:237] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } }
I20260928 02:43:43.317685 1222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e636fd848eee449b830a70ea3145c1bc. Latest consensus state: current_term: 1 leader_uuid: "e636fd848eee449b830a70ea3145c1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } } }
I20260928 02:43:43.317763 1222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:43.317698 1223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e636fd848eee449b830a70ea3145c1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e636fd848eee449b830a70ea3145c1bc" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39433 } } }
I20260928 02:43:43.317934 1223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:43.317974 1228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:43.318799 1228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:43.320600 1228 catalog_manager.cc:1347] Generated new cluster ID: 88fed0fdb3944c74b1f45c215da5631c
I20260928 02:43:43.320655 1228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:43.325348 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.313790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.27.109.190:39433)
0928 02:43:43.313981 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.313990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.314023 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:43.315359 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.315365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.315417 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.315557 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.316294 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.317373 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.318507 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.319073 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.319465 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.321736 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.321740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.321740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.321750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.323561 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.323896 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.323899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.323899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.324003 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.324169 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.324169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.324170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.324454 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.324580 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.325040 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.325166 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 02:43:43.343547 1228 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:43.344087 1228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:43.348668 1228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e636fd848eee449b830a70ea3145c1bc: Generated new TSK 0
I20260928 02:43:43.348845 1228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:43:43.377292 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:43.459502 1243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:43.459689 1243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:43.459712 1243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:43.460762 1243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:43.460795 1243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:43.460820 1243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:43.462275 1243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:43.462450 1243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:43.462692 1243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:43.467308 1248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.467314 1249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.467379 1251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:43.467643 1243 server_base.cc:1034] running on GCE node
I20260928 02:43:43.467866 1243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:43.468115 1243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:43.469270 1243 hybrid_clock.cc:648] HybridClock initialized: now 1790563423469246 us; error 36 us; skew 500 ppm
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.471719 1243 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:43:43.472077 1243 webserver.cc:460] Webserver started at http://127.27.109.129:39001/ using document root <none> and password file <none>
I20260928 02:43:43.472258 1243 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:43.472296 1243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:43.472383 1243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:43.473346 1243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "ea59bd0e48f04c0cad14523c342d3d39"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "5b2f3af97bbb6a48dec7158e41696af4"
server_key_iv: "67ce1f02ac43a7bdba3e06a182b66171"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.473629 1243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "ea59bd0e48f04c0cad14523c342d3d39"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "5b2f3af97bbb6a48dec7158e41696af4"
server_key_iv: "67ce1f02ac43a7bdba3e06a182b66171"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.474946 1243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:43.475593 1258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.475760 1243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:43.475888 1243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "ea59bd0e48f04c0cad14523c342d3d39"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "5b2f3af97bbb6a48dec7158e41696af4"
server_key_iv: "67ce1f02ac43a7bdba3e06a182b66171"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.475966 1243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:43.498459 1243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:43.500692 1243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:43.500856 1243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:43.501073 1243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:43.501355 1243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:43.501411 1243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.501451 1243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:43.501477 1243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.506989 1243 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:36475
I20260928 02:43:43.507059 1371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:36475 every 8 connection(s)
I20260928 02:43:43.507354 1243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:43.512048 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1243
I20260928 02:43:43.512161 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:43.512342 28086 external_mini_cluster.cc:1428] Setting key 710510d351914062f4ed3fa46b4340de
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:43.517900 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.508063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:46525 (local address 127.27.109.190:39433)
0928 02:43:43.508135 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.508138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.508272 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:43:43.509205 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.509206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.509219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.509270 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.509849 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.510331 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.511650 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.512052 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.512126 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.514572 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.514574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.514575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.514585 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.516329 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.516750 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.516751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.516752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.516828 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.517166 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.517167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.517167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.517304 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.517386 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.517696 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.517809 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:43:43.518096 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.508017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39433 (local address 127.27.109.129:46525)
0928 02:43:43.508157 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:43.508190 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:43.509116 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:43.509322 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:43.509326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:43.509551 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:43.509770 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.509775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.510424 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.510425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.511542 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.511544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.512148 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.512149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.512287 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.512780 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:43.512788 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:43.514443 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:43.516427 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.516432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.516434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.516667 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.516950 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.516951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.516951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.517054 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.517377 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:43.517383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:43.517551 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:43:43.517674 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:43:43.517830 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260928 02:43:43.518573 1372 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39433
I20260928 02:43:43.518676 1372 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:43.518903 1372 heartbeater.cc:511] Master 127.27.109.190:39433 requested a full tablet report, sending...
I20260928 02:43:43.519496 1181 ts_manager.cc:194] Registered new tserver with Master: ea59bd0e48f04c0cad14523c342d3d39 (127.27.109.129:36475)
I20260928 02:43:43.520418 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:46525
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:43:43.537214 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:43.616322 1377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:43.616515 1377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:43.616537 1377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:43.617712 1377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:43.617774 1377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:43.617815 1377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:43.619314 1377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:43.619577 1377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:43.619783 1377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:43.624519 1385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.624560 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:43.624699 1377 server_base.cc:1034] running on GCE node
W20260928 02:43:43.624579 1383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:43.624989 1377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:43.625219 1377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:43.626374 1377 hybrid_clock.cc:648] HybridClock initialized: now 1790563423626358 us; error 36 us; skew 500 ppm
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.628778 1377 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:43:43.629172 1377 webserver.cc:460] Webserver started at http://127.27.109.130:37099/ using document root <none> and password file <none>
I20260928 02:43:43.629388 1377 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:43.629456 1377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:43.629601 1377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:43.630623 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "72dfad458df4491890d95fd4c21ed0bd"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "409848200f5aac11c170f8b8b8ea49f9"
server_key_iv: "93af3a869279711ba41ee8aa11bdfdb5"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.630980 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "72dfad458df4491890d95fd4c21ed0bd"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "409848200f5aac11c170f8b8b8ea49f9"
server_key_iv: "93af3a869279711ba41ee8aa11bdfdb5"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.632205 1377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:43.632872 1392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.633046 1377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.633139 1377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "72dfad458df4491890d95fd4c21ed0bd"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "409848200f5aac11c170f8b8b8ea49f9"
server_key_iv: "93af3a869279711ba41ee8aa11bdfdb5"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.633212 1377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:43.646049 1377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:43.647817 1377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:43.647975 1377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:43.648178 1377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:43.648456 1377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:43.648509 1377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.648541 1377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:43.648568 1377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.653950 1377 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:39887
I20260928 02:43:43.654022 1505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:39887 every 8 connection(s)
I20260928 02:43:43.654312 1377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:43.661461 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1377
I20260928 02:43:43.661544 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:43.661734 28086 external_mini_cluster.cc:1428] Setting key 6ab2620a2570863beb5ad29292c063d3
I20260928 02:43:43.663133 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.655064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:52705 (local address 127.27.109.190:39433)
0928 02:43:43.655104 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.655106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.655294 (+ 188us) server_negotiation.cc:407] Connection header received
0928 02:43:43.656174 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.656175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.656183 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.656233 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.656766 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.657247 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.658198 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.658377 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.658426 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.660450 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.660451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.660452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.660458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.661709 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.662159 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.662160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.662161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.662214 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.662499 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.662500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.662500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.662615 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.662697 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.662960 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.663043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:43:43.663321 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.655018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39433 (local address 127.27.109.130:52705)
0928 02:43:43.655174 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:43.655213 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:43.656092 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:43.656292 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:43.656295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:43.656529 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:43.656698 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.656702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.657339 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.657340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.658124 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.658127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.658446 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.658446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.658627 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.659038 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:43.659047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:43.660358 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:43.661799 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.661806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.661808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.662090 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.662279 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.662279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.662280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.662411 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.662689 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:43.662695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:43.662827 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:43:43.662939 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:43:43.663089 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 02:43:43.663746 1506 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39433
I20260928 02:43:43.663854 1506 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:43.664036 1506 heartbeater.cc:511] Master 127.27.109.190:39433 requested a full tablet report, sending...
I20260928 02:43:43.664424 1181 ts_manager.cc:194] Registered new tserver with Master: 72dfad458df4491890d95fd4c21ed0bd (127.27.109.130:39887)
I20260928 02:43:43.664788 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:52705
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:43:43.685663 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:43.765198 1511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:43.765391 1511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:43.765413 1511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:43.766498 1511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:43.766531 1511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:43.766557 1511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:43.767974 1511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:43.768151 1511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:43.768307 1511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:43.772993 1519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.773073 1516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:43.773089 1517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:43.773218 1511 server_base.cc:1034] running on GCE node
I20260928 02:43:43.773406 1511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:43.773638 1511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:43.774809 1511 hybrid_clock.cc:648] HybridClock initialized: now 1790563423774800 us; error 38 us; skew 500 ppm
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.777093 1511 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:43:43.777501 1511 webserver.cc:460] Webserver started at http://127.27.109.131:39047/ using document root <none> and password file <none>
I20260928 02:43:43.777734 1511 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:43.777802 1511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:43.777947 1511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:43.779116 1511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "bda23a7cc7e94ccca08729dff5118f67"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d22b1f5bb79fdfae852c956d4ff82b30"
server_key_iv: "9a0d0486b86727f564c93b8cc37ae48c"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.779532 1511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "bda23a7cc7e94ccca08729dff5118f67"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d22b1f5bb79fdfae852c956d4ff82b30"
server_key_iv: "9a0d0486b86727f564c93b8cc37ae48c"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.780827 1511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:43.781562 1526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.781796 1511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:43:43.781894 1511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "bda23a7cc7e94ccca08729dff5118f67"
format_stamp: "Formatted at 2026-09-28 02:43:43 on dist-test-slave-b5p5"
server_key: "d22b1f5bb79fdfae852c956d4ff82b30"
server_key_iv: "9a0d0486b86727f564c93b8cc37ae48c"
server_key_version: "encryptionkey@0"
I20260928 02:43:43.781977 1511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:43.795619 1511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:43.797349 1511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:43.797502 1511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:43.797708 1511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:43.798014 1511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:43.798046 1511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.798076 1511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:43.798095 1511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:43.803635 1511 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39209
I20260928 02:43:43.803699 1639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39209 every 8 connection(s)
I20260928 02:43:43.804004 1511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:43.809842 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1511
I20260928 02:43:43.809921 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:43:43.810117 28086 external_mini_cluster.cc:1428] Setting key f80135719db5f584af06bf4765d2011a
I20260928 02:43:43.813452 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.804738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43383 (local address 127.27.109.190:39433)
0928 02:43:43.804876 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.804879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.805036 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:43:43.805967 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.805968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.805981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.806039 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.806700 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.807186 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.808165 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.808353 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.808435 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.810679 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.810680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.810680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.810687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.812049 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.812458 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.812460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.812460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.812555 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.812845 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.812846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.812846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.812956 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.813029 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.813281 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.813354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 02:43:43.813622 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.804638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39433 (local address 127.27.109.131:43383)
0928 02:43:43.804901 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:43.804942 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:43.805854 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:43.806100 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:43.806104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:43.806354 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:43.806603 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.806608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.807284 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.807285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.808066 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:43.808068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.808443 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.808444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:43.808576 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.809116 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:43.809123 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:43.810576 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:43.812168 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.812174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.812177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.812383 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.812608 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:43.812608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:43.812609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:43.812769 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:43.813022 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:43.813028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:43.813157 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 02:43:43.813260 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:43:43.813405 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260928 02:43:43.814028 1640 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39433
I20260928 02:43:43.814098 1640 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:43.814276 1640 heartbeater.cc:511] Master 127.27.109.190:39433 requested a full tablet report, sending...
I20260928 02:43:43.814656 1181 ts_manager.cc:194] Registered new tserver with Master: bda23a7cc7e94ccca08729dff5118f67 (127.27.109.131:39209)
I20260928 02:43:43.815043 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:43383
I20260928 02:43:43.821719 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/test-admin.keytab.
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.836176 28086 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:43:43.837678 1647 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:43.836926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.27.109.129:36475)
0928 02:43:43.837093 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.837099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.837110 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:43.837138 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.837145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.837157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.837236 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.837591 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58988: BlockingRecv error: recv got EOF from 127.0.0.1:58988 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
W20260928 02:43:43.837786 28086 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.838757 28086 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.839617 28086 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:43.846714 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.840082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.27.109.129:36475)
0928 02:43:43.840132 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.840133 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.840154 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:43:43.840201 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.840202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.840210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.840256 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.840601 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.841315 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.842017 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.842296 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.842345 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.844274 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.844281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.844281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.844287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.845815 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.846099 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.846102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.846102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.846151 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.846292 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.846292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.846292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.846392 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.846555 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.846595 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.846650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 02:43:43.848233 1647 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:43.847807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.27.109.129:36475)
0928 02:43:43.847841 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.847843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.847875 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:43:43.847918 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.847918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.847925 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.847969 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.848173 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59006: BlockingRecv error: recv got EOF from 127.0.0.1:59006 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 02:43:43.848349 28086 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.849344 28086 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.850169 28086 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:43.856674 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.850701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.27.109.129:36475)
0928 02:43:43.850754 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.850756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.850819 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:43:43.850877 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.850878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.850887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.850954 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.851301 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.851767 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.852444 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.852637 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.852725 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.854594 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.854596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.854596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.854602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.855722 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.856036 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.856037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.856037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.856091 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.856247 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.856248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.856248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.856325 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.856388 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.856471 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.856530 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 02:43:43.858062 1647 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:43:43.857542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.27.109.129:36475)
0928 02:43:43.857575 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.857577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.857630 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:43:43.857688 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.857688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.857696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.857731 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.858005 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59026: BlockingRecv error: recv got EOF from 127.0.0.1:59026 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:43:43.858104 28086 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563399140758-28086-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.859047 28086 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:43.859827 28086 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1136](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:43:43.865916 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:43.860291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.27.109.129:36475)
0928 02:43:43.860324 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:43.860326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:43.860359 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:43:43.860403 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:43.860403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:43.860410 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:43.860451 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:43.860736 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.861220 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.861882 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:43.862073 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:43.862123 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:43.863939 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:43.863940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:43.863940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:43.863945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:43.865076 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.865352 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.865353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.865353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.865397 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:43.865540 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:43.865540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:43.865540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:43.865608 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:43.865665 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:43.865746 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:43:43.865794 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:43:43.866573 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1243
I20260928 02:43:43.870898 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1377
I20260928 02:43:43.874981 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1511
I20260928 02:43:43.879160 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1149
2026-09-28T02:43:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1829 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1660](info): set up 2 sockets
Sep 28 02:43:43 dist-test-slave-b5p5 krb5kdc[1660](info): commencing operation
krb5kdc: starting...
W20260928 02:43:44.915118 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:43:44 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:43:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab.
Entry for principal oryx/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab.
I20260928 02:43:45.031510 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46325
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab
--principal=oryx/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:46325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:45.110441 1673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:45.110684 1673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:45.110719 1673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:45.111852 1673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:45.111920 1673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:45.111948 1673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:45.111976 1673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:45.112007 1673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:45.113583 1673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:46325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab
--principal=oryx/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:45.113854 1673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:45.114078 1673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:45.118862 1678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.118855 1681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.118989 1673 server_base.cc:1034] running on GCE node
W20260928 02:43:45.118856 1679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.119349 1673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.119583 1673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.120718 1673 hybrid_clock.cc:648] HybridClock initialized: now 1790563425120707 us; error 30 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.123067 1673 init.cc:377] Logged in from keytab as oryx/127.27.109.190@KRBTEST.COM (short username oryx)
I20260928 02:43:45.123415 1673 webserver.cc:460] Webserver started at http://127.27.109.190:35255/ using document root <none> and password file <none>
I20260928 02:43:45.123653 1673 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.123720 1673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.123849 1673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:45.124918 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "b8289534b4fb4ccaa0acb53188440b96"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "b817db646346daeaeb47dc829a384caf"
server_key_iv: "f9f93845ba301a36312439956d5c572e"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.125293 1673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "b8289534b4fb4ccaa0acb53188440b96"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "b817db646346daeaeb47dc829a384caf"
server_key_iv: "f9f93845ba301a36312439956d5c572e"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.126616 1673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:43:45.127310 1688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.127482 1673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:45.127578 1673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "b8289534b4fb4ccaa0acb53188440b96"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "b817db646346daeaeb47dc829a384caf"
server_key_iv: "f9f93845ba301a36312439956d5c572e"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.127669 1673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.143035 1673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.144795 1673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:45.144944 1673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.149150 1673 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:46325
I20260928 02:43:45.149199 1740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:46325 every 8 connection(s)
I20260928 02:43:45.149538 1673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:43:45.150604 1741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:45.152755 1741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Bootstrap starting.
I20260928 02:43:45.153326 1741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:45.153610 1741 log.cc:824] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:45.154286 1741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: No bootstrap required, opened a new log
I20260928 02:43:45.155784 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1673
I20260928 02:43:45.155911 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:43:45.156090 28086 external_mini_cluster.cc:1428] Setting key 923df14e496cf0c0c16df6a8b0126685
I20260928 02:43:45.156066 1741 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } }
I20260928 02:43:45.156234 1741 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:45.156296 1741 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8289534b4fb4ccaa0acb53188440b96, State: Initialized, Role: FOLLOWER
I20260928 02:43:45.156443 1741 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } }
I20260928 02:43:45.156579 1741 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:43:45.156627 1741 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:43:45.156682 1741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:45.157397 1741 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } }
I20260928 02:43:45.157508 1741 leader_election.cc:302] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8289534b4fb4ccaa0acb53188440b96; no voters:
I20260928 02:43:45.157661 1741 leader_election.cc:288] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:43:45.157680 1746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:45.157783 1746 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 LEADER]: Becoming Leader. State: Replica: b8289534b4fb4ccaa0acb53188440b96, State: Running, Role: LEADER
I20260928 02:43:45.157858 1746 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } }
I20260928 02:43:45.157927 1741 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:45.158174 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } }
I20260928 02:43:45.158241 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:45.158378 1746 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8289534b4fb4ccaa0acb53188440b96. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } }
I20260928 02:43:45.158447 1746 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:45.158951 1750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:43:45.159458 1750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:43:45.161080 1750 catalog_manager.cc:1347] Generated new cluster ID: acbb44234fff4b739d1c57cdffe6c507
I20260928 02:43:45.161134 1750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:43:45.166759 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.156572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.27.109.190:46325)
0928 02:43:45.156759 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.156769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.156858 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:43:45.157280 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.157288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.157361 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.157520 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.158282 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.159688 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.160700 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.161213 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.161335 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.163592 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.163596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.163596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.163606 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.165271 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.165640 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.165642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.165643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.165736 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.165943 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.165944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.165945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.166170 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.166346 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.166411 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.166578 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260928 02:43:45.170492 1750 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:43:45.170964 1750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:43:45.176661 1750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Generated new TSK 0
I20260928 02:43:45.176824 1750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
I20260928 02:43:45.206648 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:45.301358 1767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:45.301577 1767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:45.301612 1767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:45.303018 1767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:45.303081 1767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:45.303122 1767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:45.304508 1767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:45.304770 1767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:45.304999 1767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:45.309873 1775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.309904 1772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.309904 1773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.310286 1767 server_base.cc:1034] running on GCE node
I20260928 02:43:45.310524 1767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.310770 1767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.311998 1767 hybrid_clock.cc:648] HybridClock initialized: now 1790563425311973 us; error 39 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.314673 1767 init.cc:377] Logged in from keytab as oryx/127.27.109.129@KRBTEST.COM (short username oryx)
I20260928 02:43:45.315142 1767 webserver.cc:460] Webserver started at http://127.27.109.129:35429/ using document root <none> and password file <none>
I20260928 02:43:45.315375 1767 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.315444 1767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.315574 1767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:45.316561 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "9e29df0d061440e9ac5332a2983ccf51"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "7251f0edfbaa4d67d7a060c85c312203"
server_key_iv: "29d5d3380eca8b4f8651dc914cbc2103"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.316923 1767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "9e29df0d061440e9ac5332a2983ccf51"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "7251f0edfbaa4d67d7a060c85c312203"
server_key_iv: "29d5d3380eca8b4f8651dc914cbc2103"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.318214 1767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:43:45.319070 1782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.319267 1767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:43:45.319378 1767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9e29df0d061440e9ac5332a2983ccf51"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "7251f0edfbaa4d67d7a060c85c312203"
server_key_iv: "29d5d3380eca8b4f8651dc914cbc2103"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.319473 1767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.331702 1767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.333922 1767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:45.334079 1767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.334311 1767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:45.334679 1767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:45.334733 1767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.334766 1767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:45.334795 1767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.340481 1767 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:38723
I20260928 02:43:45.340528 1895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:38723 every 8 connection(s)
I20260928 02:43:45.340821 1767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:45.341601 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1767
I20260928 02:43:45.341713 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:43:45.341933 28086 external_mini_cluster.cc:1428] Setting key 587bdac7d180674dfd8a4ae2761b0829
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.109.130@KRBTEST.COM; defaulting to no policy
I20260928 02:43:45.351364 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.341417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:44263 (local address 127.27.109.190:46325)
0928 02:43:45.341504 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.341507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.341751 (+ 244us) server_negotiation.cc:407] Connection header received
0928 02:43:45.342423 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.342424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.342436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.342536 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.343808 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.344515 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.345533 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.345885 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.345960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.348327 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.348328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.348329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.348337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.349943 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.350324 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.350326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.350326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.350386 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.350712 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.350713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.350714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.350861 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.350944 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.351229 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.351301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:43:45.351509 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.341394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:44263)
0928 02:43:45.341606 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.341653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.342322 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.342612 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.342616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.343426 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.343696 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.343702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.344646 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.344647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.345426 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.345428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.345969 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.345970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.346063 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.346601 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.346619 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.348230 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.350031 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.350037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.350038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.350242 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.350460 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.350461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.350461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.350610 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.350944 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.350951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.351093 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.351179 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.351293 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
I20260928 02:43:45.352109 1896 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:45.352192 1896 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:45.352394 1896 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:45.352998 1705 ts_manager.cc:194] Registered new tserver with Master: 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.130@KRBTEST.COM" created.
I20260928 02:43:45.353722 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:44263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
I20260928 02:43:45.369376 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:45.451311 1901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:45.451496 1901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:45.451519 1901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:45.452560 1901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:45.452594 1901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:45.452620 1901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:45.453943 1901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:45.454108 1901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:45.454276 1901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:45.459038 1907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.459244 1901 server_base.cc:1034] running on GCE node
W20260928 02:43:45.459036 1906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.459043 1909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.459525 1901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.459794 1901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.460942 1901 hybrid_clock.cc:648] HybridClock initialized: now 1790563425460929 us; error 30 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.463310 1901 init.cc:377] Logged in from keytab as oryx/127.27.109.130@KRBTEST.COM (short username oryx)
I20260928 02:43:45.463750 1901 webserver.cc:460] Webserver started at http://127.27.109.130:41711/ using document root <none> and password file <none>
I20260928 02:43:45.463981 1901 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.464047 1901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.464192 1901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:45.465252 1901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "30e5cd985789467e87e37f8edd3ae4af"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "d73e2978c67a4c2d7c3e5cbfcc203188"
server_key_iv: "94988923a9780d339af99827fac6385d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.465600 1901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "30e5cd985789467e87e37f8edd3ae4af"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "d73e2978c67a4c2d7c3e5cbfcc203188"
server_key_iv: "94988923a9780d339af99827fac6385d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.466952 1901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:43:45.467633 1916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.467819 1901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:45.467926 1901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "30e5cd985789467e87e37f8edd3ae4af"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "d73e2978c67a4c2d7c3e5cbfcc203188"
server_key_iv: "94988923a9780d339af99827fac6385d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.468001 1901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.478330 1901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.480031 1901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:45.480182 1901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.480396 1901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:45.480669 1901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:45.480722 1901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.480762 1901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:45.480799 1901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.486518 1901 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:34557
I20260928 02:43:45.486562 2029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:34557 every 8 connection(s)
I20260928 02:43:45.486848 1901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:43:45.493814 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 1901
I20260928 02:43:45.493959 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:43:45.494161 28086 external_mini_cluster.cc:1428] Setting key fd140352ec50660756147695e60a1ba2
I20260928 02:43:45.496965 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.487530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:39491 (local address 127.27.109.190:46325)
0928 02:43:45.487578 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.487579 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.487750 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:43:45.488220 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.488221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.488230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.488287 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.489111 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.489599 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.490580 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.490767 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.490829 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.492717 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.492719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.492719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.492726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.494283 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.494801 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.494802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.494803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.494868 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.496304 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.496305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.496306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.496459 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.496538 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.496759 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.496856 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:43:45.496974 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.487497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:39491)
0928 02:43:45.487640 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.487679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.488133 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.488336 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.488339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.488862 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.489035 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.489039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.489674 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.489674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.490485 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.490488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.490873 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.490874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.490970 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.491364 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.491372 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.492616 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.494378 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.494384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.494386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.494714 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.496118 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.496119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.496119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.496214 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.496532 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.496536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.496636 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.496715 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.496825 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260928 02:43:45.497439 2030 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:45.497529 2030 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:45.497697 2030 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:45.498085 1705 ts_manager.cc:194] Registered new tserver with Master: 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
I20260928 02:43:45.498415 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:39491
WARNING: no policy specified for oryx/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
I20260928 02:43:45.519464 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:45.599743 2035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:45.599931 2035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:45.599953 2035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:45.601004 2035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:45.601040 2035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:45.601066 2035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:45.602368 2035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:45.602599 2035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:45.602771 2035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:43:45.607672 2035 server_base.cc:1034] running on GCE node
W20260928 02:43:45.607641 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.607599 2040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.607599 2041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.608017 2035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.608249 2035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.609404 2035 hybrid_clock.cc:648] HybridClock initialized: now 1790563425609394 us; error 45 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.611881 2035 init.cc:377] Logged in from keytab as oryx/127.27.109.131@KRBTEST.COM (short username oryx)
I20260928 02:43:45.612293 2035 webserver.cc:460] Webserver started at http://127.27.109.131:43171/ using document root <none> and password file <none>
I20260928 02:43:45.612481 2035 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.612522 2035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.612604 2035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:45.613556 2035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "796b727c966f4742bbe4809eb30fe17c"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "eca657a8b5495de32fd69d7d85a331ea"
server_key_iv: "50d58e0c95590d68259c328f17cd238d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.613996 2035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "796b727c966f4742bbe4809eb30fe17c"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "eca657a8b5495de32fd69d7d85a331ea"
server_key_iv: "50d58e0c95590d68259c328f17cd238d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.615314 2035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:43:45.615993 2050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.616173 2035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:45.616256 2035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "796b727c966f4742bbe4809eb30fe17c"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "eca657a8b5495de32fd69d7d85a331ea"
server_key_iv: "50d58e0c95590d68259c328f17cd238d"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.616329 2035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.625690 2035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.627945 2035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:45.628094 2035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.628293 2035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:45.628571 2035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:45.628602 2035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.628621 2035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:45.628665 2035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.634447 2035 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:35633
I20260928 02:43:45.634534 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:35633 every 8 connection(s)
I20260928 02:43:45.634868 2035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:43:45.643637 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.635545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:52031 (local address 127.27.109.190:46325)
0928 02:43:45.635584 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.635586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.635758 (+ 172us) server_negotiation.cc:407] Connection header received
0928 02:43:45.636280 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.636282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.636295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.636357 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.637226 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.637726 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.638760 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.638960 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.639044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.640884 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.640885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.640885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.640892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.642281 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.642702 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.642703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.642704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.642767 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.643015 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.643016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.643017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.643131 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.643211 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.643472 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.643550 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:43:45.643776 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2035
I20260928 02:43:45.643807 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.635513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:52031)
0928 02:43:45.635638 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.635680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.636163 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.636420 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.636424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.636961 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.637127 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.637131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.637828 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.637829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.638673 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.638675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.639051 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.639052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.639169 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.639545 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.639552 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.640790 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.642353 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.642358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.642359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.642622 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.642842 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.642844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.642844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.642946 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.643201 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.643207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.643343 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.643448 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.643589 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260928 02:43:45.643889 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:43:45.644090 28086 external_mini_cluster.cc:1428] Setting key c68c7d829f6377c905fcb757af891bc0
I20260928 02:43:45.644230 2164 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:45.644335 2164 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:45.644554 2164 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:45.644950 1705 ts_manager.cc:194] Registered new tserver with Master: 796b727c966f4742bbe4809eb30fe17c (127.27.109.131:35633)
I20260928 02:43:45.645318 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:52031
I20260928 02:43:45.655771 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.189.keytab.
Entry for principal oryx/127.27.109.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.189.keytab.
I20260928 02:43:45.680863 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:38843
--webserver_interface=127.27.109.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.189.keytab
--principal=oryx/127.27.109.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:46325,127.27.109.189:38843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:45.760506 2169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:45.760730 2169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:45.760766 2169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:45.761883 2169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:45.761947 2169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:45.761974 2169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:45.762007 2169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:45.762030 2169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:45.763468 2169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:46325,127.27.109.189:38843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.189.keytab
--principal=oryx/127.27.109.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:38843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
W20260928 02:43:45.763593 2169 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.109.190:46325,127.27.109.189:38843'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:43:45.763751 2169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:45.763963 2169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:45.768616 2175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.768622 2174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.768790 2169 server_base.cc:1034] running on GCE node
W20260928 02:43:45.768643 2177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.769070 2169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.769299 2169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.770434 2169 hybrid_clock.cc:648] HybridClock initialized: now 1790563425770414 us; error 34 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.772948 2169 init.cc:377] Logged in from keytab as oryx/127.27.109.189@KRBTEST.COM (short username oryx)
I20260928 02:43:45.773365 2169 webserver.cc:460] Webserver started at http://127.27.109.189:43831/ using document root <none> and password file <none>
I20260928 02:43:45.773597 2169 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.773661 2169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.773808 2169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:45.774899 2169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/instance:
uuid: "addd0cc386854113bc86a10a11131f99"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "66ed3311568ab5c7a25f9d600ca61059"
server_key_iv: "f1a8243117cf2a39b00f1f7125d97145"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.775242 2169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal/instance:
uuid: "addd0cc386854113bc86a10a11131f99"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "66ed3311568ab5c7a25f9d600ca61059"
server_key_iv: "f1a8243117cf2a39b00f1f7125d97145"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.776530 2169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:45.777240 2184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.777410 2169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:45.777503 2169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "addd0cc386854113bc86a10a11131f99"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "66ed3311568ab5c7a25f9d600ca61059"
server_key_iv: "f1a8243117cf2a39b00f1f7125d97145"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.777581 2169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.809432 2169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.811733 2169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:45.811915 2169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.816068 2169 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.189:38843
I20260928 02:43:45.816114 2236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.189:38843 every 8 connection(s)
I20260928 02:43:45.816468 2169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
I20260928 02:43:45.817366 2237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:45.818923 2237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.189@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:43:45.825508 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2169
I20260928 02:43:45.827541 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.819567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.27.109.190:46325)
0928 02:43:45.819645 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.819647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.819813 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:43:45.820245 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.820246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.820255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.820306 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.820834 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.821310 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.822322 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.822561 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.822615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.824542 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.824543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.824543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.824549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.825974 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.826512 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.826514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.826514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.826572 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.826845 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.826845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.826845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.826978 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.827030 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.827373 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.827453 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:43:45.827715 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.819520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38910)
0928 02:43:45.819694 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.819728 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.820165 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.820357 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.820360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.820579 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.820765 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.820769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.821400 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.821401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.822244 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.822246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.822628 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.822628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.822747 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.823144 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.823151 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.824458 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.826053 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.826059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.826061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.826411 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.826636 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.826637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.826637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.826777 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.827043 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.827050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.827216 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.827345 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.827492 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260928 02:43:45.828270 2237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.189@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
I20260928 02:43:45.835410 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.825984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.27.109.189:38843)
0928 02:43:45.826123 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.826132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.826207 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:43:45.826279 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.826280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.826290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.826346 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.827173 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.828017 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.829022 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.829340 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.829392 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.832003 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.832009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.832009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.832015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.834175 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.834577 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.834580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.834581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.834641 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.834837 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.834838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.834839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.835013 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.835163 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.835255 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.835315 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260928 02:43:45.836288 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.828515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46460)
0928 02:43:45.828572 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.828582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.828622 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.828825 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.828826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.828933 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.828973 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.828974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.829719 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.829720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.830406 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.830408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.830921 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.830922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.831013 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.831346 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.831351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.832515 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.835174 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.835175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.835175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.835399 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.835683 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.835684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.835684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.835741 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.835970 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.835971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.836006 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.836092 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.836180 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:43:45.836388 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.828600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.27.109.189:38843)
0928 02:43:45.828720 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.828722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.828727 (+ 5us) server_negotiation.cc:407] Connection header received
0928 02:43:45.828741 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.828742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.828749 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.828808 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.829073 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.829632 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.830503 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.830840 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.830912 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.832616 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.832617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.832618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.832625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.835101 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.835566 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.835567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.835568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.835615 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.835803 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.835804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.835805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.835905 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.835978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.836114 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.836230 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260928 02:43:45.837455 2237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Bootstrap starting.
I20260928 02:43:45.838027 2237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:45.838291 2237 log.cc:824] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:45.839115 2237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: No bootstrap required, opened a new log
I20260928 02:43:45.840407 2237 raft_consensus.cc:348] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } }
I20260928 02:43:45.840521 2237 raft_consensus.cc:374] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:45.840543 2237 raft_consensus.cc:729] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: addd0cc386854113bc86a10a11131f99, State: Initialized, Role: FOLLOWER
I20260928 02:43:45.840667 2237 consensus_queue.cc:260] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } }
I20260928 02:43:45.840948 2237 sys_catalog.cc:564] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:45.840934 2244 sys_catalog.cc:455] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } } }
I20260928 02:43:45.841084 2244 sys_catalog.cc:458] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:43:45.842576 2255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:45.842811 2255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:45.844115 2169 master_runner.cc:428] Detected that this master addd0cc386854113bc86a10a11131f99 is joining an existing cluster
I20260928 02:43:45.844164 2169 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.189:38843
I20260928 02:43:45.852111 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.845181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.27.109.190:46325)
0928 02:43:45.845228 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.845230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.845340 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:43:45.845463 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.845465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.845475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.845540 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.845924 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.846693 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.847654 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.847875 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.847951 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.849469 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.849470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.849471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.849477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.850931 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851329 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.851330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.851331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.851385 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851603 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.851603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.851604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.851715 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.851773 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.851958 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.852024 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:43:45.852133 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.845126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38912)
0928 02:43:45.845282 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.845292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.845407 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.845563 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.845564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.845772 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.845844 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.845846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.846772 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.846773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.847578 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.847580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.848000 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.848001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.848056 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.848636 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.848640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.849123 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.851031 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.851032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.851249 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.851476 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.851477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.851545 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.851798 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.851799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.851848 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.851930 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.852024 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260928 02:43:45.852252 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.845370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.27.109.189:38843)
0928 02:43:45.845409 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.845411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.845563 (+ 152us) server_negotiation.cc:407] Connection header received
0928 02:43:45.845630 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.845630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.845639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.845689 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.845989 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.846442 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.847501 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.847846 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.847920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.849214 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.849215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.849215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.849222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.851011 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851430 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.851431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.851431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.851482 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851702 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.851703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.851704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.851839 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.851883 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.852093 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.852167 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260928 02:43:45.852483 2262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.845332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46466)
0928 02:43:45.845496 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.845503 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.845580 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.845690 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.845691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.845874 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.845933 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.845934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.846574 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.846575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.847412 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.847414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.847928 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.847929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.848004 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.848556 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.848560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.849123 (+ 563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.851096 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.851097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.851359 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.851561 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.851561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.851562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.851627 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.851935 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.851935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.851977 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.852315 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.852391 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.190@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
I20260928 02:43:45.861914 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.854734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46474)
0928 02:43:45.854886 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.854929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.855001 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.855124 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.855127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.855445 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.855494 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.855497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.856136 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.856137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.857059 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.857063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.857446 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.857447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.857541 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.857934 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.857939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.859332 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.860829 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.860835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.860838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.861030 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.861244 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.861245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.861245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.861307 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.861554 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.861560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.861608 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.861709 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.861822 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260928 02:43:45.861917 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.854812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.27.109.189:38843)
0928 02:43:45.854848 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.854850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.854985 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:43:45.855069 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.855070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.855076 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.855119 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.855566 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.856059 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.857143 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.857383 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.857428 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.859460 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.859461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.859461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.859467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.860745 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.861119 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.861120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.861120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.861170 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.861385 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.861386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.861387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.861488 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.861560 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.861746 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.861818 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:45.862571 1705 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.189:38843: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "addd0cc386854113bc86a10a11131f99"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.189"
port: 38843
}
attrs {
promote: true
}
}
dest_uuid: "b8289534b4fb4ccaa0acb53188440b96"
cas_config_opid_index: -1
I20260928 02:43:45.862870 1705 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } }
I20260928 02:43:45.863310 2265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER addd0cc386854113bc86a10a11131f99 (127.27.109.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } }
I20260928 02:43:45.863600 2211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:45.864408 2265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.189:38843
I20260928 02:43:45.864490 2211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Refusing update from remote peer b8289534b4fb4ccaa0acb53188440b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:43:45.864493 2265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } }
I20260928 02:43:45.864549 2265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:45.864837 2169 master.cc:537] Master@127.27.109.189:38843 shutting down...
I20260928 02:43:45.865295 2265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:43:45.866123 1689 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:43:45.866541 1689 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 -> Peer addd0cc386854113bc86a10a11131f99 (127.27.109.189:38843): Couldn't send request to peer addd0cc386854113bc86a10a11131f99. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:43:45.868193 2169 raft_consensus.cc:2229] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:45.868285 2169 raft_consensus.cc:2258] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:45.868336 2169 tablet_replica.cc:330] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: stopping tablet replica
I20260928 02:43:45.881203 2169 master.cc:552] Master@127.27.109.189:38843 shutdown complete.
I20260928 02:43:45.881269 2169 master_runner.cc:305] Clearing existing system tablet
I20260928 02:43:45.881399 2169 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.881460 2169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.882061 2169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260928 02:43:45.882544 2269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.882682 2169 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.882742 2169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.882769 2169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "addd0cc386854113bc86a10a11131f99"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "66ed3311568ab5c7a25f9d600ca61059"
server_key_iv: "f1a8243117cf2a39b00f1f7125d97145"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.883359 2169 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:45.884740 2169 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:43:45.884809 2169 log.cc:1196] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:43:45.885108 2169 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Deleting consensus metadata
I20260928 02:43:45.885232 2169 master_runner.cc:315] Copying system tablet from 127.27.109.190:46325
I20260928 02:43:45.885783 2169 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.190:46325
I20260928 02:43:45.896449 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.886156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.27.109.190:46325)
0928 02:43:45.886200 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.886202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.890681 (+ 4479us) server_negotiation.cc:407] Connection header received
0928 02:43:45.890744 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.890745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.890755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.890808 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.891154 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.891632 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.892670 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.892841 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.892890 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.893861 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.893863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.893863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.893871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.895345 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.895726 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.895726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.895727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.895774 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.895995 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.895996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.895996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.896121 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.896173 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.896328 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.896389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:45.896518 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.886088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38926)
0928 02:43:45.890595 (+ 4507us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:45.890608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:45.890678 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:45.890813 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:45.890814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:45.891020 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:45.891086 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.891088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.891708 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.891709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.892591 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:45.892593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.892909 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.892910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:45.892977 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.893395 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:45.893399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:45.893778 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:45.895436 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.895438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.895438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.895652 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.895836 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:45.895837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:45.895837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:45.895901 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:45.896187 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:45.896188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:45.896234 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:43:45.896312 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:43:45.896408 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4486,"thread_start_us":66,"threads_started":1}
I20260928 02:43:45.896801 1725 tablet_copy_service.cc:140] P b8289534b4fb4ccaa0acb53188440b96: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer addd0cc386854113bc86a10a11131f99 ({username='oryx', principal='oryx/127.27.109.189@KRBTEST.COM'} at 127.0.0.1:38926)
I20260928 02:43:45.896898 1725 tablet_copy_service.cc:161] P b8289534b4fb4ccaa0acb53188440b96: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer addd0cc386854113bc86a10a11131f99 at {username='oryx', principal='oryx/127.27.109.189@KRBTEST.COM'} at 127.0.0.1:38926: session id = addd0cc386854113bc86a10a11131f99-00000000000000000000000000000000
I20260928 02:43:45.897673 1725 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:43:45.898241 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:43:45.899164 2169 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: tablet copy: Starting download of 0 data blocks...
I20260928 02:43:45.899315 2169 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: tablet copy: Starting download of 1 WAL segments...
I20260928 02:43:45.900172 2169 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:43:45.900887 1725 tablet_copy_service.cc:342] P b8289534b4fb4ccaa0acb53188440b96: Request end of tablet copy session addd0cc386854113bc86a10a11131f99-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.189@KRBTEST.COM'} at 127.0.0.1:38926
I20260928 02:43:45.900943 1725 tablet_copy_service.cc:433] P b8289534b4fb4ccaa0acb53188440b96: ending tablet copy session addd0cc386854113bc86a10a11131f99-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer addd0cc386854113bc86a10a11131f99
I20260928 02:43:45.903380 2169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:45.904510 2276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.904544 2279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:45.904670 2277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:45.904834 2169 server_base.cc:1034] running on GCE node
I20260928 02:43:45.904930 2169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:45.905051 2169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:45.906172 2169 hybrid_clock.cc:648] HybridClock initialized: now 1790563425906163 us; error 22 us; skew 500 ppm
Sep 28 02:43:45 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:45.907477 2169 init.cc:377] Logged in from keytab as oryx/127.27.109.189@KRBTEST.COM (short username oryx)
I20260928 02:43:45.907742 2169 webserver.cc:460] Webserver started at http://127.27.109.189:33183/ using document root <none> and password file <none>
I20260928 02:43:45.907858 2169 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:45.907884 2169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:45.908464 2169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:45.908954 2286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:45.909070 2169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:45.909117 2169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "addd0cc386854113bc86a10a11131f99"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "66ed3311568ab5c7a25f9d600ca61059"
server_key_iv: "f1a8243117cf2a39b00f1f7125d97145"
server_key_version: "encryptionkey@0"
I20260928 02:43:45.909171 2169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:45.926321 2169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:45.927831 2169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:45.930930 2169 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.189:38843
I20260928 02:43:45.930948 2338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.189:38843 every 8 connection(s)
I20260928 02:43:45.931299 2169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
I20260928 02:43:45.932169 2339 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:43:45.932672 2339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Bootstrap starting.
I20260928 02:43:45.935477 2339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:43:45.935699 2339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: Bootstrap complete.
I20260928 02:43:45.935892 2339 raft_consensus.cc:348] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } }
I20260928 02:43:45.935981 2339 raft_consensus.cc:729] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: addd0cc386854113bc86a10a11131f99, State: Initialized, Role: LEARNER
I20260928 02:43:45.936021 2339 consensus_queue.cc:260] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } }
I20260928 02:43:45.936197 2339 sys_catalog.cc:564] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:45.936220 2342 sys_catalog.cc:455] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } }
I20260928 02:43:45.936331 2342 sys_catalog.cc:458] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:43:45.937729 2354 catalog_manager.cc:1259] Loaded cluster ID: acbb44234fff4b739d1c57cdffe6c507
I20260928 02:43:45.938733 2354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: loading cluster ID for follower catalog manager: success
I20260928 02:43:45.940341 2354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: acquiring CA information for follower catalog manager: success
I20260928 02:43:45.940711 2354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:43:45.943506 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:45.936801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.27.109.189:38843)
0928 02:43:45.936947 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:45.936949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:45.936956 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:45.936977 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:45.936978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:45.936987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:45.937052 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:45.937749 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.938380 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.939382 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:45.939582 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:45.939662 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:45.940799 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:45.940800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:45.940801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:45.940807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:45.942336 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.942727 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.942728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.942729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.942784 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:45.942993 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:45.942994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:45.942995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:45.943105 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:45.943206 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:45.943317 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:43:45.943411 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.188.keytab.
Entry for principal oryx/127.27.109.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.188.keytab.
I20260928 02:43:45.969844 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:46045
--webserver_interface=127.27.109.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.188.keytab
--principal=oryx/127.27.109.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:43:46.049340 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:46.049544 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:46.049568 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:46.050683 2357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:43:46.050721 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:46.050735 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:46.050747 2357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:43:46.050760 2357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:43:46.052132 2357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.188.keytab
--principal=oryx/127.27.109.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:46.052300 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:46.052454 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:46.057170 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.057281 2357 server_base.cc:1034] running on GCE node
W20260928 02:43:46.057180 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.057180 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.057649 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:46.057978 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:46.059150 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790563426059146 us; error 35 us; skew 500 ppm
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:46.061543 2357 init.cc:377] Logged in from keytab as oryx/127.27.109.188@KRBTEST.COM (short username oryx)
I20260928 02:43:46.061949 2357 webserver.cc:460] Webserver started at http://127.27.109.188:42961/ using document root <none> and password file <none>
I20260928 02:43:46.062175 2357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.062242 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.062391 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:43:46.063388 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/instance:
uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
format_stamp: "Formatted at 2026-09-28 02:43:46 on dist-test-slave-b5p5"
server_key: "20a7d8fd521cb8f2f0a075f1698a1aa5"
server_key_iv: "d821b0c72fbf35694088f61b41702b93"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.063730 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal/instance:
uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
format_stamp: "Formatted at 2026-09-28 02:43:46 on dist-test-slave-b5p5"
server_key: "20a7d8fd521cb8f2f0a075f1698a1aa5"
server_key_iv: "d821b0c72fbf35694088f61b41702b93"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.064978 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:46.065713 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.065886 2357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:46.065979 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
format_stamp: "Formatted at 2026-09-28 02:43:46 on dist-test-slave-b5p5"
server_key: "20a7d8fd521cb8f2f0a075f1698a1aa5"
server_key_iv: "d821b0c72fbf35694088f61b41702b93"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.066056 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.077229 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:46.079548 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:46.079710 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:46.083405 2357 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.188:46045
I20260928 02:43:46.083454 2424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.188:46045 every 8 connection(s)
I20260928 02:43:46.083760 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
I20260928 02:43:46.084131 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2357
I20260928 02:43:46.085229 2425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:46.087141 2425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.188@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:43:46.096851 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.088007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.27.109.190:46325)
0928 02:43:46.088057 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.088059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.088102 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:43:46.088173 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.088174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.088184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.088239 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.088635 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.089167 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.089927 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.090274 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.090357 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.093334 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.093335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.093335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.093342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.095262 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.095728 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.095730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.095730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.095794 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.096030 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.096032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.096032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.096204 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.096283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.096653 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.096740 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:43:46.097077 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.087831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38928)
0928 02:43:46.088010 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.088051 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.088115 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.088251 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.088252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.088475 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.088533 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.088535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.089262 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.089263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.089841 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.089843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.090390 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.090391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.090437 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.090977 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.090986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.092586 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.095345 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.095351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.095353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.095613 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.095874 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.095874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.095875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.095959 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.096358 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.096364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.096465 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.096591 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.096810 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260928 02:43:46.098125 2425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:43:46.098340 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.084781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.27.109.188:46045)
0928 02:43:46.084949 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.084965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.084997 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:43:46.085581 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.085589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.085645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.085799 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.086362 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.087701 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.088584 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.089008 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.089098 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.091952 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.091967 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.091967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.091978 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.094041 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.094454 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.094457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.094457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.094579 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.095139 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.095140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.095140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.097611 (+ 2471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.097750 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.098167 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.098238 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.188@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
I20260928 02:43:46.108877 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.098396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46520)
0928 02:43:46.100925 (+ 2529us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.100938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.100995 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.101145 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.101146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.101272 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.101313 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.101315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.101983 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.101985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.102786 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.102788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.103281 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.103282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.103371 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.103752 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.103757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.105219 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.107683 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.107683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.107684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.107919 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.108162 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.108163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.108163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.108252 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.108540 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.108541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.108596 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.108677 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.108782 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2511}
I20260928 02:43:46.109056 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.098542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.27.109.189:38843)
0928 02:43:46.098609 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.098612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.100983 (+ 2371us) server_negotiation.cc:407] Connection header received
0928 02:43:46.101062 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.101063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.101074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.101129 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.101388 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.101899 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.102878 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.103165 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.103231 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.105292 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.105293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.105293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.105317 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.107582 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.108031 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.108033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.108033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.108099 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.108327 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.108327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.108328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.108483 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.108555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.108763 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.108851 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:43:46.109436 2425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 46045 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.188@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
I20260928 02:43:46.116834 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.109734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.0.0.1:49328)
0928 02:43:46.109778 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.109789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.109822 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.109945 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.109946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.110082 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.110117 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.110118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.110872 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.110873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.111522 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.111524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.111946 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.111946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.112013 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.112406 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.112421 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.113832 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.115633 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.115634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.115634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.115863 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.116105 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.116106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.116106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.116173 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.116470 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.116471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.116515 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.116589 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.116689 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:43:46.116916 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.109761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.27.109.188:46045)
0928 02:43:46.109807 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.109809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.109818 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:43:46.109874 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.109875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.109885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.109941 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.110191 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.110798 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.111595 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.111853 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.111945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.113926 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.113927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.113928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.113934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.115534 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.115956 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.115957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.115958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.116031 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.116244 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.116245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.116246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.116403 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.116488 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.116611 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.116743 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":28}
I20260928 02:43:46.118637 2425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Bootstrap starting.
I20260928 02:43:46.119685 2425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:46.119958 2425 log.cc:824] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:46.120853 2425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: No bootstrap required, opened a new log
I20260928 02:43:46.122797 2425 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 46045 } }
I20260928 02:43:46.122972 2425 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:46.123014 2425 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47f0d0bda0a4bce94e2c6d494fef06f, State: Initialized, Role: FOLLOWER
I20260928 02:43:46.123162 2425 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 46045 } }
I20260928 02:43:46.123425 2430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 46045 } } }
I20260928 02:43:46.123499 2425 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:46.123512 2430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:43:46.125409 2441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:43:46.125475 2441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:43:46.127754 2357 master_runner.cc:428] Detected that this master c47f0d0bda0a4bce94e2c6d494fef06f is joining an existing cluster
I20260928 02:43:46.127799 2357 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.188:46045
I20260928 02:43:46.135865 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.128996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.27.109.188:46045)
0928 02:43:46.129030 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.129031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.129068 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:43:46.129159 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.129160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.129169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.129198 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.130026 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.130803 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.131899 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.132165 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.132200 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.133230 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.133232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.133232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.133239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.134731 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.135096 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.135097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.135098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.135151 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.135350 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.135351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.135352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.135461 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.135533 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.135679 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.135752 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:43:46.136169 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.128816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.0.0.1:49338)
0928 02:43:46.129010 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.129021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.129091 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.129258 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.129259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.129524 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.129941 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.129944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.130909 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.130909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.131808 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.131810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.132271 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.132272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.132333 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.132833 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.132837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.133141 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.134823 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.134824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.134825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.135021 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.135224 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.135226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.135226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.135280 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.135524 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.135525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.135576 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.135958 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.136082 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":44,"thread_start_us":76,"threads_started":1}
I20260928 02:43:46.149950 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1767
I20260928 02:43:46.155730 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:38723
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:46325
--webserver_port=35429
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:46.159510 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.129120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.27.109.189:38843)
0928 02:43:46.129157 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.129159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.129379 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:43:46.129480 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.129481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.129491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.129521 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.130032 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.130792 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.131915 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.132207 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.132244 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.156165 (+ 23921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.156167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.156168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.156178 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.158058 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.158596 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.158598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.158598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.158659 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.158855 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.158856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.158857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.159010 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.159053 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.159261 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.159365 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:46.160276 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.129199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46536)
0928 02:43:46.129325 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.129334 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.129411 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.129593 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.129593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.129823 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.129949 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.129951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.130909 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.130909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.131808 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.131810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.132309 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.132309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.132375 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.155199 (+ 22824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.155206 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.155685 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.158177 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.158179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.158179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.158510 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.158727 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.158728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.158728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.158788 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.159100 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.159101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.159149 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.159986 (+ 837us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.160121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":7,"thread_start_us":76,"threads_started":1}
I20260928 02:43:46.161859 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.128869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38932)
0928 02:43:46.129323 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.129333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.129414 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.129593 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.129593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.154823 (+ 25230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.154905 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.154908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.155930 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.155930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.156807 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.156809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.157330 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.157330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.157395 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.158299 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.158304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.158768 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.160695 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.160696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.160696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.160900 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.161121 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.161121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.161122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.161175 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.161524 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.161525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.161585 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.161661 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.161755 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":89,"threads_started":1}
I20260928 02:43:46.161929 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.128830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.27.109.190:46325)
0928 02:43:46.128875 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.128877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.129388 (+ 511us) server_negotiation.cc:407] Connection header received
0928 02:43:46.129490 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.129491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.129502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.129531 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.155038 (+ 25507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.155826 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.156904 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.157214 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.157254 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.158867 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.158868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.158869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.158876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.160583 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.160996 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.160997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.160997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.161050 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.161258 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.161260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.161260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.161434 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.161523 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.161711 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.161797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563425, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.190@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
I20260928 02:43:46.172268 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.164337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.27.109.188:46045)
0928 02:43:46.164401 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.164403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.164413 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:46.164447 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.164448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.164460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.164527 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.164830 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.165595 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.166820 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.167160 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.167242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.169430 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.169432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.169432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.169439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.171058 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.171455 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.171456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.171456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.171511 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.171750 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.171751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.171751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.171876 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.171940 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.172110 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.172183 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:46.172268 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.164162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.0.0.1:49342)
0928 02:43:46.164212 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.164224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.164265 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.164540 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.164541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.164697 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.164738 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.164740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.165683 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.165685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.166738 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.166741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.167251 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.167252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.167329 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.167796 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.167800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.169335 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.171138 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.171139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.171140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.171382 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.171581 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.171582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.171582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.171658 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.171945 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.171946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.172001 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.172078 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.172170 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:43:46.173250 1705 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.188:46045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.188"
port: 46045
}
attrs {
promote: true
}
}
dest_uuid: "b8289534b4fb4ccaa0acb53188440b96"
cas_config_opid_index: 5
I20260928 02:43:46.173480 1705 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:46.173774 2265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c47f0d0bda0a4bce94e2c6d494fef06f (127.27.109.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.174669 2265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.188:46045
I20260928 02:43:46.174737 2265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.174804 2265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:46.175407 2399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:46.175559 2357 master.cc:537] Master@127.27.109.188:46045 shutting down...
I20260928 02:43:46.176429 2399 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 1 FOLLOWER]: Refusing update from remote peer b8289534b4fb4ccaa0acb53188440b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:43:46.176702 2264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:43:46.177455 1691 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 -> Peer c47f0d0bda0a4bce94e2c6d494fef06f (127.27.109.188:46045): Couldn't send request to peer c47f0d0bda0a4bce94e2c6d494fef06f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:43:46.179054 2357 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:46.179134 2357 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:46.179174 2357 tablet_replica.cc:330] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: stopping tablet replica
I20260928 02:43:46.182530 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.174975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.0.0.1:46538)
0928 02:43:46.178544 (+ 3569us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.178557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.178606 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.178906 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.178908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.178910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.178946 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.178948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.179951 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.179952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.181120 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.181123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.182077 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.182078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.182143 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.182168 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.182398 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.182437 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3552}
I20260928 02:43:46.182819 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.175079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.27.109.189:38843)
0928 02:43:46.175146 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.175149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.178659 (+ 3510us) server_negotiation.cc:407] Connection header received
0928 02:43:46.178685 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.178686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.178705 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.178755 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.179029 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.179800 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.181220 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.181802 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.181872 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.181934 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.182714 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.182757 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:43:46.182996 2357 master.cc:552] Master@127.27.109.188:46045 shutdown complete.
I20260928 02:43:46.183038 2357 master_runner.cc:305] Clearing existing system tablet
I20260928 02:43:46.183166 2357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.183207 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.183285 2313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 LEARNER]: Refusing update from remote peer b8289534b4fb4ccaa0acb53188440b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:43:46.183459 2264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:43:46.183934 2313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c47f0d0bda0a4bce94e2c6d494fef06f (127.27.109.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.185194 2342 sys_catalog.cc:455] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8289534b4fb4ccaa0acb53188440b96. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.185271 2342 sys_catalog.cc:458] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:43:46.185304 2342 sys_catalog.cc:455] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: NON_VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: true } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.185343 2342 sys_catalog.cc:458] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:43:46.185537 2264 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: attempting to promote NON_VOTER addd0cc386854113bc86a10a11131f99 to VOTER
I20260928 02:43:46.185703 2264 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:46.186272 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.186082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.0.0.1:49352)
0928 02:43:46.186198 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.186240 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:46045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260928 02:43:46.186312 2313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 LEARNER]: Refusing update from remote peer b8289534b4fb4ccaa0acb53188440b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:43:46.186667 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.186521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.0.0.1:49356)
0928 02:43:46.186620 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.186642 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:46045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
I20260928 02:43:46.186659 2265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 02:43:46.186903 1691 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 -> Peer c47f0d0bda0a4bce94e2c6d494fef06f (127.27.109.188:46045): Couldn't send request to peer c47f0d0bda0a4bce94e2c6d494fef06f. Status: Network error: Client connection negotiation failed: client connection to 127.27.109.188:46045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:43:46.187408 2265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, addd0cc386854113bc86a10a11131f99 (127.27.109.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.188234 2265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.188326 2265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:46.188387 2265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.188431 2265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:43:46.188510 2357 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260928 02:43:46.189018 2313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, addd0cc386854113bc86a10a11131f99 (127.27.109.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.189074 2457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.189164 2357 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.189198 2357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.189217 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
format_stamp: "Formatted at 2026-09-28 02:43:46 on dist-test-slave-b5p5"
server_key: "20a7d8fd521cb8f2f0a075f1698a1aa5"
server_key_iv: "d821b0c72fbf35694088f61b41702b93"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.189853 2342 sys_catalog.cc:455] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8289534b4fb4ccaa0acb53188440b96" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.189922 2342 sys_catalog.cc:458] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:43:46.189889 2357 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:46.191179 2357 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:43:46.191248 2357 log.cc:1196] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:43:46.191447 2357 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Deleting consensus metadata
I20260928 02:43:46.191601 2357 master_runner.cc:315] Copying system tablet from 127.27.109.190:46325
I20260928 02:43:46.192180 2357 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.190:46325
I20260928 02:43:46.201339 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.192600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.27.109.190:46325)
0928 02:43:46.192641 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.192643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.194663 (+ 2020us) server_negotiation.cc:407] Connection header received
0928 02:43:46.194714 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.194714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.194725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.194778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.195134 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.195886 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.196948 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.197254 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.197324 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.198398 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.198399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.198399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.198406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.200153 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.200542 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.200543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.200544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.200602 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.200810 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.200811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.200812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.200939 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.201014 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.201174 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.201248 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:46.201364 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.192427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:38934)
0928 02:43:46.194563 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.194575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.194656 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.194799 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.194800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.195002 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.195054 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.195056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.195971 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.195972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.196859 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.196861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.197336 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.197337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.197392 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.197910 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.197915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.198295 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.200237 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.200238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.200238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.200460 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.200674 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.200674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.200675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.200741 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.201015 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.201016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.201061 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.201134 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.201229 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2116,"thread_start_us":68,"threads_started":1}
I20260928 02:43:46.201881 1725 tablet_copy_service.cc:140] P b8289534b4fb4ccaa0acb53188440b96: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c47f0d0bda0a4bce94e2c6d494fef06f ({username='oryx', principal='oryx/127.27.109.188@KRBTEST.COM'} at 127.0.0.1:38934)
I20260928 02:43:46.201934 1725 tablet_copy_service.cc:161] P b8289534b4fb4ccaa0acb53188440b96: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c47f0d0bda0a4bce94e2c6d494fef06f at {username='oryx', principal='oryx/127.27.109.188@KRBTEST.COM'} at 127.0.0.1:38934: session id = c47f0d0bda0a4bce94e2c6d494fef06f-00000000000000000000000000000000
I20260928 02:43:46.202524 1725 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:43:46.203013 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:43:46.204141 2357 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: tablet copy: Starting download of 0 data blocks...
I20260928 02:43:46.204265 2357 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: tablet copy: Starting download of 1 WAL segments...
I20260928 02:43:46.205144 2357 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:43:46.206066 1725 tablet_copy_service.cc:342] P b8289534b4fb4ccaa0acb53188440b96: Request end of tablet copy session c47f0d0bda0a4bce94e2c6d494fef06f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.188@KRBTEST.COM'} at 127.0.0.1:38934
I20260928 02:43:46.206116 1725 tablet_copy_service.cc:433] P b8289534b4fb4ccaa0acb53188440b96: ending tablet copy session c47f0d0bda0a4bce94e2c6d494fef06f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c47f0d0bda0a4bce94e2c6d494fef06f
I20260928 02:43:46.208735 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:46.209853 2464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.211352 2465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.211570 2357 server_base.cc:1034] running on GCE node
W20260928 02:43:46.211601 2467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.211841 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:46.211984 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:46.213088 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790563426213080 us; error 22 us; skew 500 ppm
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:46.214504 2357 init.cc:377] Logged in from keytab as oryx/127.27.109.188@KRBTEST.COM (short username oryx)
I20260928 02:43:46.214742 2357 webserver.cc:460] Webserver started at http://127.27.109.188:43515/ using document root <none> and password file <none>
I20260928 02:43:46.214982 2357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.215024 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.215724 2357 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:46.216308 2474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.216472 2357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:46.216524 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "c47f0d0bda0a4bce94e2c6d494fef06f"
format_stamp: "Formatted at 2026-09-28 02:43:46 on dist-test-slave-b5p5"
server_key: "20a7d8fd521cb8f2f0a075f1698a1aa5"
server_key_iv: "d821b0c72fbf35694088f61b41702b93"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.216578 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.236229 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:46.238258 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:46.241381 2357 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.188:46045
I20260928 02:43:46.241417 2526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.188:46045 every 8 connection(s)
I20260928 02:43:46.241745 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
I20260928 02:43:46.243302 2527 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:43:46.243852 2527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Bootstrap starting.
I20260928 02:43:46.246824 2527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:43:46.247051 2527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: Bootstrap complete.
I20260928 02:43:46.247253 2527 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:46.247345 2527 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c47f0d0bda0a4bce94e2c6d494fef06f, State: Initialized, Role: LEARNER
I20260928 02:43:46.247381 2527 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:46.247546 2527 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:43:46.247552 2530 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } } }
I20260928 02:43:46.247768 2530 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f [sys.catalog]: This master's current role is: LEARNER
I20260928 02:43:46.249809 2541 catalog_manager.cc:1259] Loaded cluster ID: acbb44234fff4b739d1c57cdffe6c507
I20260928 02:43:46.249861 2541 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: loading cluster ID for follower catalog manager: success
I20260928 02:43:46.250958 2541 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: acquiring CA information for follower catalog manager: success
I20260928 02:43:46.251245 2541 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c47f0d0bda0a4bce94e2c6d494fef06f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:43:46.262010 2450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:46.262184 2450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:46.262207 2450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:46.263447 2450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:46.263489 2450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:46.263515 2450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:43:46.265029 2450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:38723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=35429
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:46.265255 2450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:46.265492 2450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:46.270219 2546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.270227 2547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.270506 2549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.270607 2450 server_base.cc:1034] running on GCE node
I20260928 02:43:46.270780 2450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:46.271037 2450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:46.272212 2450 hybrid_clock.cc:648] HybridClock initialized: now 1790563426272196 us; error 38 us; skew 500 ppm
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:46.274749 2450 init.cc:377] Logged in from keytab as oryx/127.27.109.129@KRBTEST.COM (short username oryx)
I20260928 02:43:46.275157 2450 webserver.cc:460] Webserver started at http://127.27.109.129:35429/ using document root <none> and password file <none>
I20260928 02:43:46.275394 2450 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.275475 2450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.276810 2450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:46.277460 2556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.277673 2450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:43:46.277779 2450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9e29df0d061440e9ac5332a2983ccf51"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "7251f0edfbaa4d67d7a060c85c312203"
server_key_iv: "29d5d3380eca8b4f8651dc914cbc2103"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.278111 2450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.290581 2450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:46.292614 2450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:46.292768 2450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:46.292982 2450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:46.293262 2450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:46.293313 2450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.293352 2450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:46.293380 2450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.299095 2450 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:38723
I20260928 02:43:46.299134 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:38723 every 8 connection(s)
I20260928 02:43:46.299477 2450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:43:46.301570 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2450
I20260928 02:43:46.301728 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1901
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
I20260928 02:43:46.308735 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:34557
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:46325
--webserver_port=41711
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:43:46.312105 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:34393 (local address 127.27.109.189:38843)
0928 02:43:46.300409 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.300412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.300587 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:43:46.301047 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.301048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.301060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.301128 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.302587 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303381 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.304957 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305306 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305346 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.308713 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.308714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.308714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.308721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.310446 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.310963 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.310965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.310965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.311031 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311311 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.311312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.311313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.311474 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.311519 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.311869 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.311960 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260928 02:43:46.312588 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:35723 (local address 127.27.109.188:46045)
0928 02:43:46.300720 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.300722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.300730 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:46.301047 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.301048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.301060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.301128 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.302192 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303295 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.304946 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305241 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305297 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.309117 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.309118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.309118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.309125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.311074 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311473 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.311474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.311474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.311528 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311720 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.311721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.311721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.311843 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.311902 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.312407 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.312496 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":50,"threads_started":1}
I20260928 02:43:46.313019 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:56877 (local address 127.27.109.190:46325)
0928 02:43:46.300671 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.300673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.301371 (+ 698us) server_negotiation.cc:407] Connection header received
0928 02:43:46.301405 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.301406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.301417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.301480 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.302546 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303291 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.304956 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305247 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305297 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.310073 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.310075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.310075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.310083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.311804 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.312236 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.312237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.312238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.312305 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.312498 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.312499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.312499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.312618 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.312665 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.312861 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.312936 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:46.313284 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.27.109.129:34393)
0928 02:43:46.300529 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.300539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.300946 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.301576 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.301577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.302393 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.302487 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.302490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303475 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.303476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.304848 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.304851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305415 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.305509 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.306256 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.306261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.308569 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.310561 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.310567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.310570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.310872 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.311105 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311105 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.311106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.311236 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.311567 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.311573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.311728 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.312229 (+ 501us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.313028 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":756,"spinlock_wait_cycles":384,"thread_start_us":110,"threads_started":1}
I20260928 02:43:46.313436 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:56877)
0928 02:43:46.301270 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.301280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.301357 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.301519 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.301520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.302355 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.302415 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.302417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303420 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.303421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.304849 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.304852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305370 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.305508 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.306179 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.306189 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.309651 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.311950 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.311951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.312142 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.312374 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.312375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.312375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.312427 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.312718 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.312719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.312756 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.313138 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.313237 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":116,"thread_start_us":215,"threads_started":1}
I20260928 02:43:46.313532 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.300145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.27.109.129:35723)
0928 02:43:46.300327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.300360 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.300937 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.301210 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.301215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.301844 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.302090 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.302095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.303420 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.303421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.304849 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.304852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.305359 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.305360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.305508 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.306388 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.306393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.309024 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.311167 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.311169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.311392 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.311597 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.311598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.311598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.311655 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.311904 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.311905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.311947 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.312037 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.312957 (+ 920us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":123,"thread_start_us":90,"threads_started":1}
I20260928 02:43:46.313928 2672 heartbeater.cc:347] Connected to a master server at 127.27.109.189:38843
I20260928 02:43:46.314039 2672 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.314112 2670 heartbeater.cc:347] Connected to a master server at 127.27.109.188:46045
I20260928 02:43:46.314186 2670 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.314239 2672 heartbeater.cc:511] Master 127.27.109.189:38843 requested a full tablet report, sending...
I20260928 02:43:46.314271 2670 heartbeater.cc:511] Master 127.27.109.188:46045 requested a full tablet report, sending...
I20260928 02:43:46.314430 2671 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:46.314493 2671 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.314561 2671 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:46.314759 2484 ts_manager.cc:194] Registered new tserver with Master: 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723)
I20260928 02:43:46.314949 2303 ts_manager.cc:194] Registered new tserver with Master: 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723)
I20260928 02:43:46.315239 1705 ts_manager.cc:194] Re-registered known tserver with Master: 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723)
I20260928 02:43:46.315788 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:56877
W20260928 02:43:46.391976 2678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:46.392168 2678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:46.392190 2678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:46.393294 2678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:46.393327 2678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:46.393352 2678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:43:46.394830 2678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:34557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=41711
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:46.395008 2678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:46.395169 2678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:46.399971 2686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.399976 2683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.399992 2684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.400523 2678 server_base.cc:1034] running on GCE node
I20260928 02:43:46.400744 2678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:46.400991 2678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:46.402155 2678 hybrid_clock.cc:648] HybridClock initialized: now 1790563426402142 us; error 24 us; skew 500 ppm
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:46.404718 2678 init.cc:377] Logged in from keytab as oryx/127.27.109.130@KRBTEST.COM (short username oryx)
I20260928 02:43:46.405169 2678 webserver.cc:460] Webserver started at http://127.27.109.130:41711/ using document root <none> and password file <none>
I20260928 02:43:46.405395 2678 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.405478 2678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.406880 2678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:43:46.407445 2693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.407622 2678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:43:46.407727 2678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "30e5cd985789467e87e37f8edd3ae4af"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "d73e2978c67a4c2d7c3e5cbfcc203188"
server_key_iv: "94988923a9780d339af99827fac6385d"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.408056 2678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.420995 2678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:46.423189 2678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:46.423345 2678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:46.423555 2678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:46.423830 2678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:46.423882 2678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.423921 2678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:46.423949 2678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.429785 2678 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:34557
I20260928 02:43:46.429837 2806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:34557 every 8 connection(s)
I20260928 02:43:46.430153 2678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:43:46.434286 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2678
I20260928 02:43:46.434455 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2035
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
I20260928 02:43:46.440303 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:35633
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:46325
--webserver_port=43171
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
I20260928 02:43:46.441823 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.430871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:50223 (local address 127.27.109.190:46325)
0928 02:43:46.430913 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.430915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.431154 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:43:46.431729 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.431731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.431743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.431823 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.432467 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433245 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434261 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.434617 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434672 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.437651 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.437653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.437653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.437661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.439423 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.440181 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.440182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.440182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.440240 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.440801 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.440802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.440802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.440937 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.440980 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.441635 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.441723 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:46.442104 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.431205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:58717 (local address 127.27.109.189:38843)
0928 02:43:46.431250 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.431252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.431335 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:43:46.431754 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.431756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.431768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.431822 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.432579 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433282 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434119 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.434386 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434442 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.438269 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.438270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.438271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.438278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.440838 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.441240 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.441241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.441242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.441298 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.441490 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.441491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.441492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.441641 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.441707 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.441901 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.441986 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:43:46.442708 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.430865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:50223)
0928 02:43:46.431016 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.431055 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.431628 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.431897 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.431900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.432187 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.432375 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.432379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433331 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.433332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.434169 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.434172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.434719 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.434785 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.435793 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.435798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.437544 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.439503 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.439510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.439512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.439824 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.440331 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.440332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.440333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.440718 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.441028 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.441034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.441196 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.442459 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.442580 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":27,"thread_start_us":87,"threads_started":1}
I20260928 02:43:46.442690 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.431149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.27.109.130:58717)
0928 02:43:46.431284 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.431293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.431651 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.432284 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.432284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.432446 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.432501 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.432503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433377 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.433378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.434010 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.434012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.434489 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.434491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.434666 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.435341 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.435350 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.438182 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.440930 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.440931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.440931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.441161 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.441365 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.441366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.441366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.441434 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.441715 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.441716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.441763 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.442239 (+ 476us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.442436 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260928 02:43:46.443207 2808 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:46.443219 2809 heartbeater.cc:347] Connected to a master server at 127.27.109.189:38843
I20260928 02:43:46.443320 2808 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.443320 2809 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.443531 2809 heartbeater.cc:511] Master 127.27.109.189:38843 requested a full tablet report, sending...
I20260928 02:43:46.443531 2808 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:46.443929 1705 ts_manager.cc:194] Re-registered known tserver with Master: 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
I20260928 02:43:46.444309 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:50223
I20260928 02:43:46.444535 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.430831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:48401 (local address 127.27.109.188:46045)
0928 02:43:46.430875 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.430912 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.431131 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:43:46.431754 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.431756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.431768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.431812 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.432486 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433005 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.435895 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.436216 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.436289 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.441028 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.441029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.441029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.441036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.442950 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.443650 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.443652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.443652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.443707 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.443906 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.443906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.443907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.444036 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.444074 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.444302 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.444393 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:43:46.444546 2303 ts_manager.cc:194] Registered new tserver with Master: 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
I20260928 02:43:46.445043 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.430865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.27.109.130:48401)
0928 02:43:46.431015 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.431056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.431651 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.431935 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.431936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.432203 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.432377 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.432383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.433099 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.433100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.435813 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.435815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.438986 (+ 3171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.438986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.439035 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.439420 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.439424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.440931 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.443214 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.443215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.443215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.443526 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.443775 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.443776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.443776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.443842 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.444155 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.444156 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.444197 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.444719 (+ 522us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.444857 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":4,"spinlock_wait_cycles":7424,"thread_start_us":93,"threads_started":1}
I20260928 02:43:46.445510 2807 heartbeater.cc:347] Connected to a master server at 127.27.109.188:46045
I20260928 02:43:46.445580 2807 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.445658 2807 heartbeater.cc:511] Master 127.27.109.188:46045 requested a full tablet report, sending...
I20260928 02:43:46.445992 2484 ts_manager.cc:194] Registered new tserver with Master: 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
W20260928 02:43:46.522266 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:43:46.522456 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:43:46.522537 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:43:46.523957 2814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:43:46.524011 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:43:46.524039 2814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:43:46.525532 2814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:35633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=43171
--tserver_master_addrs=127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:43:46.525780 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:43:46.525978 2814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:43:46.530859 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.530892 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:43:46.530877 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:43:46.531127 2814 server_base.cc:1034] running on GCE node
I20260928 02:43:46.531477 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:43:46.531674 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:43:46.532846 2814 hybrid_clock.cc:648] HybridClock initialized: now 1790563426532834 us; error 31 us; skew 500 ppm
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:43:46.535369 2814 init.cc:377] Logged in from keytab as oryx/127.27.109.131@KRBTEST.COM (short username oryx)
I20260928 02:43:46.535884 2814 webserver.cc:460] Webserver started at http://127.27.109.131:43171/ using document root <none> and password file <none>
I20260928 02:43:46.536160 2814 fs_manager.cc:359] Metadata directory not provided
I20260928 02:43:46.536244 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:43:46.537667 2814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:43:46.538244 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.538405 2814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:43:46.538520 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "796b727c966f4742bbe4809eb30fe17c"
format_stamp: "Formatted at 2026-09-28 02:43:45 on dist-test-slave-b5p5"
server_key: "eca657a8b5495de32fd69d7d85a331ea"
server_key_iv: "50d58e0c95590d68259c328f17cd238d"
server_key_version: "encryptionkey@0"
I20260928 02:43:46.538861 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:43:46.546309 2814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:43:46.548605 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:43:46.548760 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:43:46.548976 2814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:43:46.549324 2814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:43:46.549377 2814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.549417 2814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:43:46.549443 2814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:43:46.555243 2814 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:35633
I20260928 02:43:46.555289 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:35633 every 8 connection(s)
I20260928 02:43:46.555626 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:43:46.556203 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 2814
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.189@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
Sep 28 02:43:46 dist-test-slave-b5p5 krb5kdc[1660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563426, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.188@KRBTEST.COM
I20260928 02:43:46.567232 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.557342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.27.109.188:46045)
0928 02:43:46.558247 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.558250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.558445 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:43:46.558581 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.558582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.558595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.558635 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.559563 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.560305 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561474 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561764 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561801 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.563461 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.563462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.563463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.563469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.565314 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.566109 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.566110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.566110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.566168 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.566373 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.566374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.566374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.566542 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.566638 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.567023 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.567105 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":56,"threads_started":1}
I20260928 02:43:46.568995 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.556721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:47171 (local address 127.27.109.189:38843)
0928 02:43:46.556775 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.556779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.556970 (+ 191us) server_negotiation.cc:407] Connection header received
0928 02:43:46.557459 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.557461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.557474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.557506 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.558592 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.559352 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.560930 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561223 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561259 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.565334 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.565335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.565336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.565342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.567182 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.567922 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.567923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.567923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.567978 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.568257 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.568257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.568258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.568382 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.568425 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.568790 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.568877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:43:46.569751 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.556656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:38843 (local address 127.27.109.131:47171)
0928 02:43:46.556911 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.556921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.557323 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.558026 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.558027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.558202 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.558453 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.558459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.559488 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.559489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.560834 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.560836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561333 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.561489 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.562030 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.562039 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.565238 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.567498 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.567505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.567508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.567830 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.568055 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.568057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.568057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.568177 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.568472 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.568477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.568634 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.569325 (+ 691us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.569500 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":54,"thread_start_us":63,"threads_started":1}
I20260928 02:43:46.569830 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.558280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.27.109.189:38843)
0928 02:43:46.558418 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.558419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.562232 (+ 3813us) server_negotiation.cc:407] Connection header received
0928 02:43:46.562317 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.562318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.562331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.562387 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.563038 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.563731 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.564741 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.565068 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.565152 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.566583 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.566584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.566584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.566590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.568501 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.568885 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.568885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.568886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.568935 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.569127 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.569128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.569129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.569233 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.569328 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.569659 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.569728 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260928 02:43:46.570526 2945 heartbeater.cc:347] Connected to a master server at 127.27.109.189:38843
I20260928 02:43:46.570618 2945 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.570712 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.556380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:40195 (local address 127.27.109.188:46045)
0928 02:43:46.556431 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.556433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.556680 (+ 247us) server_negotiation.cc:407] Connection header received
0928 02:43:46.557505 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.557506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.557515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.557545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.560633 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561332 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.563623 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.563903 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.563940 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.567671 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.567672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.567672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.567679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.569450 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.569844 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.569845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.569845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.569895 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.570114 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.570115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.570116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.570247 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.570297 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.570554 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.570631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:43:46.570801 2945 heartbeater.cc:511] Master 127.27.109.189:38843 requested a full tablet report, sending...
I20260928 02:43:46.571173 2303 ts_manager.cc:194] Registered new tserver with Master: 796b727c966f4742bbe4809eb30fe17c (127.27.109.131:35633)
I20260928 02:43:46.571208 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.556958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:36513)
0928 02:43:46.557219 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.557228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.557335 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.557653 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.557656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.558511 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.558573 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.558575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.559513 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.559514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.561489 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.561491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.562016 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.562017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.562080 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.563750 (+ 1670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.563755 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.567540 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.570009 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.570010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.570011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.570203 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.570410 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.570411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.570411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.570504 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.570858 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.570859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.570904 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.570995 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.571095 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":119,"thread_start_us":61,"threads_started":1}
I20260928 02:43:46.571233 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.557073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:36513 (local address 127.27.109.190:46325)
0928 02:43:46.557215 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.557217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.557272 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:43:46.557450 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.557451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.557464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.557495 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.558660 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.559397 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561575 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561877 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561918 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.567974 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.567975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.567976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.567983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.569933 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.570286 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.570287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.570287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.570339 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.570581 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.570582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.570582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.570716 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.570761 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.571060 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.571140 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":112}
I20260928 02:43:46.571336 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.556382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:46045 (local address 127.27.109.131:40195)
0928 02:43:46.556544 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.556593 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.557417 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.557654 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.557657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.558162 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:43:46.559816 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.559817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.562227 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.562228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.563533 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.563536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.564010 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.564011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.564073 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.564451 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:43:46.564456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:43:46.567156 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:43:46.569542 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.569543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.569543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.569760 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.569978 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:43:46.569978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:43:46.569979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:43:46.570041 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:43:46.570328 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:43:46.570329 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:43:46.570370 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.570434 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.570558 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 02:43:46.571923 2944 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46325
I20260928 02:43:46.571975 2944 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.571991 2943 heartbeater.cc:347] Connected to a master server at 127.27.109.188:46045
I20260928 02:43:46.572041 2944 heartbeater.cc:511] Master 127.27.109.190:46325 requested a full tablet report, sending...
I20260928 02:43:46.572048 2943 heartbeater.cc:464] Registering TS with master...
I20260928 02:43:46.572151 2943 heartbeater.cc:511] Master 127.27.109.188:46045 requested a full tablet report, sending...
I20260928 02:43:46.572261 1705 ts_manager.cc:194] Re-registered known tserver with Master: 796b727c966f4742bbe4809eb30fe17c (127.27.109.131:35633)
I20260928 02:43:46.572790 1705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:36513
I20260928 02:43:46.573113 2484 ts_manager.cc:194] Registered new tserver with Master: 796b727c966f4742bbe4809eb30fe17c (127.27.109.131:35633)
I20260928 02:43:46.574110 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.557922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.27.109.190:46325)
0928 02:43:46.558313 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.558315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.558637 (+ 322us) server_negotiation.cc:407] Connection header received
0928 02:43:46.558661 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.558662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.558677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.558738 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.559548 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.560354 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561493 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.561815 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.561853 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.571466 (+ 9613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.571467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.571468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.571475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.573144 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.573510 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.573510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.573511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.573563 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.573757 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.573757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.573758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.573841 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.573907 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.573994 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.574054 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":354,"thread_start_us":80,"threads_started":1}
I20260928 02:43:46.575457 1705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:43:46.576889 1705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:43:46.584296 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.27.109.129:38723)
0928 02:43:46.580884 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.580888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.580899 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:46.581135 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.581137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.581152 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.581248 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.581384 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.582379 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.583473 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.583932 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.584054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.584111 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.584176 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.584212 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260928 02:43:46.584308 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:38723 (local address 127.0.0.1:51566)
0928 02:43:46.580415 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.580426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.580470 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.581257 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.581258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.581260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.581300 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.581302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.582514 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.582515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.583392 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.583394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.584035 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.584036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.584098 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.584101 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.584163 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.584204 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260928 02:43:46.584355 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:34557 (local address 127.0.0.1:43060)
0928 02:43:46.580539 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.580551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.580609 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.581025 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.581026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.581028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.581071 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.581072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.582234 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.582235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.583223 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.583225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.584091 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.584092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.584153 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.584157 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.584184 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.584222 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:43:46.585039 2604 tablet_service.cc:1467] Processing CreateTablet for tablet e0ce03f25954474eab1cb5268923dce8 (DEFAULT_TABLE table=test-table [id=482d81f4118a4ecbaea0ef0d1287cd4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:46.585147 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.27.109.130:34557)
0928 02:43:46.580703 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.580708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.580719 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:43:46.580760 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.580763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.580780 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.581013 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.581127 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.582150 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.583292 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.583996 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.584723 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.584803 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.584828 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.584866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":95,"threads_started":1}
I20260928 02:43:46.585397 2604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ce03f25954474eab1cb5268923dce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:46.585788 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:35633 (local address 127.0.0.1:53090)
0928 02:43:46.580527 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.580535 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.580560 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.581511 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.581511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.581513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.581543 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.581545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.583488 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.583489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.584512 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.584514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.585552 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.585553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.585611 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.585614 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.585680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.585721 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:43:46.585815 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.580930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.27.109.131:35633)
0928 02:43:46.581358 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.581363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.581373 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:43:46.581403 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.581405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.581418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.581513 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.581594 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.582369 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.584588 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.585487 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.585570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.585614 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.585691 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.585735 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":88,"threads_started":1}
I20260928 02:43:46.585734 2741 tablet_service.cc:1467] Processing CreateTablet for tablet e0ce03f25954474eab1cb5268923dce8 (DEFAULT_TABLE table=test-table [id=482d81f4118a4ecbaea0ef0d1287cd4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:46.586051 2741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ce03f25954474eab1cb5268923dce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:46.587081 2877 tablet_service.cc:1467] Processing CreateTablet for tablet e0ce03f25954474eab1cb5268923dce8 (DEFAULT_TABLE table=test-table [id=482d81f4118a4ecbaea0ef0d1287cd4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:43:46.587381 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ce03f25954474eab1cb5268923dce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:43:46.589365 2968 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Bootstrap starting.
I20260928 02:43:46.589601 2967 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Bootstrap starting.
I20260928 02:43:46.589839 2968 tablet_bootstrap.cc:654] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:46.590066 2968 log.cc:824] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:46.590178 2967 tablet_bootstrap.cc:654] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:46.590523 2967 log.cc:824] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:46.590827 2968 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: No bootstrap required, opened a new log
I20260928 02:43:46.590855 2971 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Bootstrap starting.
I20260928 02:43:46.590909 2968 ts_tablet_manager.cc:1397] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:46.591434 2971 tablet_bootstrap.cc:654] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Neither blocks nor log segments found. Creating new log.
I20260928 02:43:46.591739 2971 log.cc:824] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:43:46.592437 2971 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: No bootstrap required, opened a new log
I20260928 02:43:46.592439 2967 tablet_bootstrap.cc:492] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: No bootstrap required, opened a new log
I20260928 02:43:46.592509 2971 ts_tablet_manager.cc:1397] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:43:46.592602 2967 ts_tablet_manager.cc:1397] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:43:46.592767 2968 raft_consensus.cc:348] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.592935 2968 raft_consensus.cc:374] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:46.592964 2968 raft_consensus.cc:729] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e5cd985789467e87e37f8edd3ae4af, State: Initialized, Role: FOLLOWER
I20260928 02:43:46.593072 2968 consensus_queue.cc:260] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.593327 2808 heartbeater.cc:503] Master 127.27.109.190:46325 was elected leader, sending a full tablet report...
I20260928 02:43:46.594362 2968 ts_tablet_manager.cc:1428] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:46.595067 2971 raft_consensus.cc:348] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.595237 2971 raft_consensus.cc:374] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:46.595279 2971 raft_consensus.cc:729] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796b727c966f4742bbe4809eb30fe17c, State: Initialized, Role: FOLLOWER
I20260928 02:43:46.595386 2971 consensus_queue.cc:260] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.595628 2971 ts_tablet_manager.cc:1428] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:43:46.595656 2944 heartbeater.cc:503] Master 127.27.109.190:46325 was elected leader, sending a full tablet report...
I20260928 02:43:46.596398 2967 raft_consensus.cc:348] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.596565 2967 raft_consensus.cc:374] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:43:46.596611 2967 raft_consensus.cc:729] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29df0d061440e9ac5332a2983ccf51, State: Initialized, Role: FOLLOWER
I20260928 02:43:46.596714 2967 consensus_queue.cc:260] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.596954 2967 ts_tablet_manager.cc:1428] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:43:46.597070 2671 heartbeater.cc:503] Master 127.27.109.190:46325 was elected leader, sending a full tablet report...
I20260928 02:43:46.620712 2974 raft_consensus.cc:482] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:43:46.620814 2974 raft_consensus.cc:504] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.621078 2974 leader_election.cc:288] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723), 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
I20260928 02:43:46.623852 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.621365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:55917 (local address 127.27.109.129:38723)
0928 02:43:46.621416 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.621419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.621427 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:46.621452 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.621453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.621461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.621546 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.621693 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.622179 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623161 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.623513 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623598 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.623607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.623755 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.623788 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:43:46.623852 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.621274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:38723 (local address 127.27.109.131:55917)
0928 02:43:46.621326 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.621338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.621378 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.621550 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.621551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.621552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.621624 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.621626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.622265 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.622266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.623069 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.623071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.623596 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.623674 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.623676 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.623738 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.623777 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260928 02:43:46.623871 2964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.621358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:51775 (local address 127.27.109.130:34557)
0928 02:43:46.621422 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.621425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.621459 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:43:46.621526 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.621527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.621536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.621611 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:43:46.621750 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.622196 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623160 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.623515 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623607 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.623615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.623768 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.623803 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:43:46.624119 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.621285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:34557 (local address 127.27.109.131:51775)
0928 02:43:46.621401 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:46.621413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:43:46.621474 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:43:46.621615 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:43:46.621615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:43:46.621617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:43:46.621673 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.621674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.622281 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.622282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.623071 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:43:46.623073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.623607 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.623608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:43:46.623691 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.623694 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:43:46.623764 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:43:46.623809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":71}
I20260928 02:43:46.624222 2624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ce03f25954474eab1cb5268923dce8" candidate_uuid: "796b727c966f4742bbe4809eb30fe17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29df0d061440e9ac5332a2983ccf51" is_pre_election: true
I20260928 02:43:46.624375 2624 raft_consensus.cc:2454] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796b727c966f4742bbe4809eb30fe17c in term 0.
I20260928 02:43:46.624616 2833 leader_election.cc:302] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 796b727c966f4742bbe4809eb30fe17c, 9e29df0d061440e9ac5332a2983ccf51; no voters:
I20260928 02:43:46.624594 2761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ce03f25954474eab1cb5268923dce8" candidate_uuid: "796b727c966f4742bbe4809eb30fe17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e5cd985789467e87e37f8edd3ae4af" is_pre_election: true
I20260928 02:43:46.624727 2761 raft_consensus.cc:2454] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 796b727c966f4742bbe4809eb30fe17c in term 0.
I20260928 02:43:46.624737 2974 raft_consensus.cc:2781] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:43:46.624790 2974 raft_consensus.cc:482] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:43:46.624809 2974 raft_consensus.cc:3037] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:46.625716 2974 raft_consensus.cc:504] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.625847 2974 leader_election.cc:288] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [CANDIDATE]: Term 1 election: Requested vote from peers 9e29df0d061440e9ac5332a2983ccf51 (127.27.109.129:38723), 30e5cd985789467e87e37f8edd3ae4af (127.27.109.130:34557)
I20260928 02:43:46.626008 2624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ce03f25954474eab1cb5268923dce8" candidate_uuid: "796b727c966f4742bbe4809eb30fe17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29df0d061440e9ac5332a2983ccf51"
I20260928 02:43:46.626020 2761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ce03f25954474eab1cb5268923dce8" candidate_uuid: "796b727c966f4742bbe4809eb30fe17c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30e5cd985789467e87e37f8edd3ae4af"
I20260928 02:43:46.626085 2761 raft_consensus.cc:3037] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:46.626103 2624 raft_consensus.cc:3037] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:43:46.626806 2761 raft_consensus.cc:2454] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796b727c966f4742bbe4809eb30fe17c in term 1.
I20260928 02:43:46.626922 2624 raft_consensus.cc:2454] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 796b727c966f4742bbe4809eb30fe17c in term 1.
I20260928 02:43:46.626971 2832 leader_election.cc:302] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30e5cd985789467e87e37f8edd3ae4af, 796b727c966f4742bbe4809eb30fe17c; no voters:
I20260928 02:43:46.627054 2974 raft_consensus.cc:2781] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:43:46.627179 2974 raft_consensus.cc:686] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 1 LEADER]: Becoming Leader. State: Replica: 796b727c966f4742bbe4809eb30fe17c, State: Running, Role: LEADER
I20260928 02:43:46.627244 2974 consensus_queue.cc:237] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } }
I20260928 02:43:46.628049 1705 catalog_manager.cc:5495] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c reported cstate change: term changed from 0 to 1, leader changed from <none> to 796b727c966f4742bbe4809eb30fe17c (127.27.109.131). New cstate: current_term: 1 leader_uuid: "796b727c966f4742bbe4809eb30fe17c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796b727c966f4742bbe4809eb30fe17c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35633 } health_report { overall_health: HEALTHY } } }
I20260928 02:43:46.641419 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.639079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.27.109.131:35633)
0928 02:43:46.639139 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.639141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.639149 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:43:46.639221 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.639221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.639243 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.639316 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:43:46.639454 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.639895 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.640635 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.640821 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.640895 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.641074 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:43:46.641176 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:43:46.641261 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.641304 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.641324 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:43:46.643069 2624 raft_consensus.cc:1261] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 1 FOLLOWER]: Refusing update from remote peer 796b727c966f4742bbe4809eb30fe17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:46.643069 2761 raft_consensus.cc:1261] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 1 FOLLOWER]: Refusing update from remote peer 796b727c966f4742bbe4809eb30fe17c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:43:46.643373 2974 consensus_queue.cc:1035] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e29df0d061440e9ac5332a2983ccf51" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:46.643523 2976 consensus_queue.cc:1035] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30e5cd985789467e87e37f8edd3ae4af" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:43:46.646418 2980 mvcc.cc:205] Tried to move back new op lower bound from 7334147795527921664 to 7334147795465928704. Current Snapshot: MvccSnapshot[applied={T|T < 7334147795527921664}]
I20260928 02:43:46.647279 2981 mvcc.cc:205] Tried to move back new op lower bound from 7334147795527921664 to 7334147795465928704. Current Snapshot: MvccSnapshot[applied={T|T < 7334147795527921664 or (T in {7334147795527921664})}]
I20260928 02:43:46.649128 1704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:46.649219 1704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:43:46.650355 1704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8289534b4fb4ccaa0acb53188440b96: Sending DeleteTablet for 3 replicas of tablet e0ce03f25954474eab1cb5268923dce8
I20260928 02:43:46.650658 2604 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ce03f25954474eab1cb5268923dce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:46 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51566
I20260928 02:43:46.650739 2741 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ce03f25954474eab1cb5268923dce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:46 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:43060
I20260928 02:43:46.650870 2989 tablet_replica.cc:330] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: stopping tablet replica
I20260928 02:43:46.650933 2877 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ce03f25954474eab1cb5268923dce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:43:46 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:53090
I20260928 02:43:46.650967 2989 raft_consensus.cc:2229] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:46.651073 2989 raft_consensus.cc:2258] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:46.651257 2992 tablet_replica.cc:330] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: stopping tablet replica
I20260928 02:43:46.651407 2990 tablet_replica.cc:330] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: stopping tablet replica
I20260928 02:43:46.651536 2990 raft_consensus.cc:2229] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:43:46.651659 2992 raft_consensus.cc:2229] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:43:46.651693 2989 ts_tablet_manager.cc:1905] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:46.651840 2990 raft_consensus.cc:2258] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:46.651927 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 1673
I20260928 02:43:46.652050 2992 raft_consensus.cc:2258] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:43:46.652526 2990 ts_tablet_manager.cc:1905] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:46.654059 2990 ts_tablet_manager.cc:1918] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:46.654181 2990 log.cc:1196] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/e0ce03f25954474eab1cb5268923dce8
I20260928 02:43:46.654213 2992 ts_tablet_manager.cc:1905] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:43:46.654572 2990 ts_tablet_manager.cc:1939] T e0ce03f25954474eab1cb5268923dce8 P 30e5cd985789467e87e37f8edd3ae4af: Deleting consensus metadata
I20260928 02:43:46.655750 2989 ts_tablet_manager.cc:1918] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:46.655836 2989 log.cc:1196] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/e0ce03f25954474eab1cb5268923dce8
I20260928 02:43:46.656075 2989 ts_tablet_manager.cc:1939] T e0ce03f25954474eab1cb5268923dce8 P 9e29df0d061440e9ac5332a2983ccf51: Deleting consensus metadata
W20260928 02:43:46.658480 2695 connection.cc:535] server connection from 127.0.0.1:43060 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:43:46.658504 2560 connection.cc:535] server connection from 127.0.0.1:51566 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:43:46.659111 2992 ts_tablet_manager.cc:1918] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:43:46.659181 2992 log.cc:1196] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/e0ce03f25954474eab1cb5268923dce8
I20260928 02:43:46.659435 2992 ts_tablet_manager.cc:1939] T e0ce03f25954474eab1cb5268923dce8 P 796b727c966f4742bbe4809eb30fe17c: Deleting consensus metadata
W20260928 02:43:46.659749 2833 connection.cc:441] server connection from 127.0.0.1:53090 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53090 (request call id 1) could send its response
I20260928 02:43:46.668262 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.651271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.27.109.188:46045)
0928 02:43:46.651372 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.651374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.659845 (+ 8471us) server_negotiation.cc:407] Connection header received
0928 02:43:46.659944 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.659945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.659954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.659983 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.660733 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.661479 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.662570 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.662861 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.662902 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.665108 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.665110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.665110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.665116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.667034 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667452 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.667506 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667704 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.667818 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.667885 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.668097 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.668168 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:43:46.668272 2959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.660174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.27.109.189:38843)
0928 02:43:46.660210 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.660212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.660219 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:43:46.660237 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.660238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.660247 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.660320 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.661061 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.661844 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.662877 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.663176 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.663214 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.664841 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.664842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.664843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.664850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.666794 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667366 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.667420 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667629 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.667773 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.667850 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.668072 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.668157 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:43:46.668465 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:46.660491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.27.109.188:46045)
0928 02:43:46.660531 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:43:46.660533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:43:46.660743 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:43:46.660840 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:43:46.660841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:43:46.660852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:43:46.660882 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:43:46.661933 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.662673 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.663599 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:43:46.663906 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:43:46.663969 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:43:46.665422 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:43:46.665423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:43:46.665423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:43:46.665429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:43:46.667270 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667650 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.667697 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:43:46.667908 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:43:46.667909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:43:46.667909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:43:46.668037 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:43:46.668100 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:43:46.668287 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:43:46.668370 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:43:47.647809 3000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:47.647440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:33009)
0928 02:43:47.647665 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:47.647746 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":53,"threads_started":1}
I20260928 02:43:47.647843 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:47.647343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:41545)
0928 02:43:47.647722 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:47.647798 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":87,"threads_started":1}
I20260928 02:43:47.648238 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:47.647868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:44641)
0928 02:43:47.648136 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:47.648197 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":48,"threads_started":1}
W20260928 02:43:47.648316 2671 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:46325 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
W20260928 02:43:47.648442 2808 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:46325 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
W20260928 02:43:47.648581 2944 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:46325 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:48.150978 3002 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:43:48.151101 3002 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:48.151327 3002 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:43:48.151643 3003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:48.151457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39424)
0928 02:43:48.151553 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:48.151605 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
W20260928 02:43:48.151829 2287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:48.152249 3003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:48.152156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39436)
0928 02:43:48.152204 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:48.152230 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:43:48.152400 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:48.152470 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:43:48.152601 3002 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:43:48.649075 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:48.648702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:45599)
0928 02:43:48.648926 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:48.648967 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":59,"threads_started":1}
I20260928 02:43:48.649405 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:48.648890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:45033)
0928 02:43:48.649320 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:48.649359 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":66,"threads_started":1}
I20260928 02:43:48.649503 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:48.648956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:46525)
0928 02:43:48.649420 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:48.649458 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":69,"threads_started":1}
I20260928 02:43:49.649981 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:49.649539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:35351)
0928 02:43:49.649869 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:49.649919 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260928 02:43:49.650319 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:49.650118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:34271)
0928 02:43:49.650232 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:49.650273 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
I20260928 02:43:49.650357 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:49.649841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:60909)
0928 02:43:49.650261 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:49.650300 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":49,"threads_started":1}
I20260928 02:43:50.092342 3011 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:43:50.092435 3011 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:50.092569 3011 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:43:50.092875 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.092692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39702)
0928 02:43:50.092799 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.092838 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260928 02:43:50.093276 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.093179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39704)
0928 02:43:50.093225 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.093253 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:43:50.093355 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:50.093429 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:43:50.093511 3011 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:43:50.650931 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.650644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:35583)
0928 02:43:50.650829 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.650872 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":53,"threads_started":1}
I20260928 02:43:50.650987 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.650763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:58485)
0928 02:43:50.650920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.650950 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":44,"threads_started":1}
I20260928 02:43:50.651226 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:50.650464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:40629)
0928 02:43:50.651133 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:50.651166 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":62,"threads_started":1}
I20260928 02:43:51.651679 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.651430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:47061)
0928 02:43:51.651571 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.651614 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 02:43:51.651976 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.651642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:57723)
0928 02:43:51.651905 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.651935 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":45,"threads_started":1}
I20260928 02:43:51.651976 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:51.651608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:51427)
0928 02:43:51.651889 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:51.651937 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":49,"threads_started":1}
I20260928 02:43:52.652781 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.652571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:38695)
0928 02:43:52.652682 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.652722 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260928 02:43:52.652820 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.652177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:48891)
0928 02:43:52.652730 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.652771 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":63,"threads_started":1}
I20260928 02:43:52.652940 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:52.652604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:43325)
0928 02:43:52.652820 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:52.652863 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":65,"threads_started":1}
I20260928 02:43:53.296365 3023 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:43:53.296458 3023 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:53.296653 3023 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:43:53.297008 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.296827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39942)
0928 02:43:53.296931 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.296969 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
W20260928 02:43:53.297092 2287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 02:43:53.297479 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.297352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:39944)
0928 02:43:53.297425 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.297457 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:43:53.297572 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:53.297639 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:43:53.297725 3023 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:43:53.653442 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.653184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:37379)
0928 02:43:53.653346 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.653386 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
I20260928 02:43:53.653580 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.653187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:58775)
0928 02:43:53.653502 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.653539 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":59,"threads_started":1}
I20260928 02:43:53.653635 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:53.653329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:57229)
0928 02:43:53.653547 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:53.653585 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":52,"threads_started":1}
I20260928 02:43:54.654239 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:54.654004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:47137)
0928 02:43:54.654131 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:54.654179 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260928 02:43:54.654312 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:54.654004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:45649)
0928 02:43:54.654236 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:54.654269 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":59,"threads_started":1}
I20260928 02:43:54.654511 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:54.653896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:59691)
0928 02:43:54.654419 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:54.654453 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":66,"threads_started":1}
I20260928 02:43:55.382946 3032 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:43:55.383035 3032 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:55.383229 3032 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:43:55.383559 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.383367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:36518)
0928 02:43:55.383475 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.383521 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260928 02:43:55.383994 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.383901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:36530)
0928 02:43:55.383944 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.383970 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:43:55.384099 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:55.384166 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:43:55.384269 3032 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:43:55.655041 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.654786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:51625)
0928 02:43:55.654946 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.654989 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260928 02:43:55.655047 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.654775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:56143)
0928 02:43:55.654967 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.654996 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260928 02:43:55.655088 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:55.654851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:32883)
0928 02:43:55.655001 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:55.655037 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 02:43:56.655766 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:56.655479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:54531)
0928 02:43:56.655646 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:56.655695 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260928 02:43:56.655890 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:56.655652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:36571)
0928 02:43:56.655770 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:56.655808 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 02:43:56.656184 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:56.655731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:50073)
0928 02:43:56.656096 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:56.656143 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":46,"threads_started":1}
I20260928 02:43:57.656535 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:57.656263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:45239)
0928 02:43:57.656420 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:57.656470 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 02:43:57.656697 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:57.656263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:42179)
0928 02:43:57.656614 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:57.656653 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":76,"threads_started":1}
I20260928 02:43:57.656726 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:57.656516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:50101)
0928 02:43:57.656646 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:57.656681 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260928 02:43:58.021704 3044 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:43:58.021808 3044 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:43:58.021955 3044 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:43:58.022265 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.022099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:36714)
0928 02:43:58.022197 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:58.022228 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260928 02:43:58.022655 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.022585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:36720)
0928 02:43:58.022614 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:58.022639 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 02:43:58.022725 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:43:58.022781 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:43:58.022866 3044 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:43:58.657214 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.656983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:58703)
0928 02:43:58.657117 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:58.657161 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 02:43:58.657269 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.657019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:51971)
0928 02:43:58.657177 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:58.657218 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260928 02:43:58.657328 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:58.657138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:33935)
0928 02:43:58.657263 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:58.657290 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":45,"threads_started":1}
I20260928 02:43:59.657863 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:59.657668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:33347)
0928 02:43:59.657771 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:59.657811 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20260928 02:43:59.657863 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:59.657623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:50045)
0928 02:43:59.657760 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:59.657803 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260928 02:43:59.657948 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:43:59.657786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:33573)
0928 02:43:59.657876 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:43:59.657904 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
I20260928 02:44:00.658566 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.658261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:57779)
0928 02:44:00.658401 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.658507 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260928 02:44:00.658583 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.658360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:50679)
0928 02:44:00.658511 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.658541 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":68,"threads_started":1}
I20260928 02:44:00.658579 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:00.658342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:46923)
0928 02:44:00.658446 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:00.658530 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260928 02:44:01.659220 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.658971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:50307)
0928 02:44:01.659120 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.659161 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260928 02:44:01.659250 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.659058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:58403)
0928 02:44:01.659175 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.659209 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260928 02:44:01.659251 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:01.658971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:42747)
0928 02:44:01.659175 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:01.659209 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":57,"threads_started":1}
I20260928 02:44:02.659969 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.659750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:53799)
0928 02:44:02.659870 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.659912 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 02:44:02.660013 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.659781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:43681)
0928 02:44:02.659939 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.659970 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
I20260928 02:44:02.660029 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:02.659781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:46517)
0928 02:44:02.659939 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:02.659977 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
I20260928 02:44:03.535622 3062 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:44:03.535723 3062 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:44:03.535948 3062 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:44:03.536310 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.536123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:43294)
0928 02:44:03.536230 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.536268 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
W20260928 02:44:03.536392 2287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 02:44:03.536741 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.536650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:43302)
0928 02:44:03.536698 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.536724 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:44:03.536820 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:44:03.536876 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:44:03.536963 3062 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:44:03.660779 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.660542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:41079)
0928 02:44:03.660690 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.660728 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260928 02:44:03.660764 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.660547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:47587)
0928 02:44:03.660661 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.660706 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260928 02:44:03.661000 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:03.660772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:57289)
0928 02:44:03.660913 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:03.660955 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":51,"threads_started":1}
I20260928 02:44:04.661396 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.661170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:56355)
0928 02:44:04.661290 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.661335 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260928 02:44:04.661445 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.661213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:49147)
0928 02:44:04.661378 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.661405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":55,"threads_started":1}
I20260928 02:44:04.661540 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:04.661328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:39121)
0928 02:44:04.661452 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:04.661489 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":43,"threads_started":1}
I20260928 02:44:05.662061 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.661819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:45317)
0928 02:44:05.661931 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.661974 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260928 02:44:05.662102 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.661858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:59567)
0928 02:44:05.662015 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.662050 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":56,"threads_started":1}
I20260928 02:44:05.662144 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:05.661970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:47547)
0928 02:44:05.662070 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:05.662102 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260928 02:44:06.662819 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.662554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:60485)
0928 02:44:06.662715 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.662760 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260928 02:44:06.662868 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.662657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:57697)
0928 02:44:06.662783 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.662821 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260928 02:44:06.663080 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:06.662646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:37719)
0928 02:44:06.663000 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:06.663034 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":43,"threads_started":1}
I20260928 02:44:07.663635 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.663393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:34537)
0928 02:44:07.663546 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.663585 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":40,"threads_started":1}
I20260928 02:44:07.663626 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.663338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:54483)
0928 02:44:07.663507 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.663570 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260928 02:44:07.663774 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:07.663611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:57463)
0928 02:44:07.663701 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:07.663730 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":42,"threads_started":1}
I20260928 02:44:08.664341 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.664121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:50765)
0928 02:44:08.664235 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.664278 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 02:44:08.664356 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.664094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:36883)
0928 02:44:08.664289 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.664319 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260928 02:44:08.664433 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:08.664191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:44741)
0928 02:44:08.664346 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:08.664381 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":51,"threads_started":1}
I20260928 02:44:09.665069 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.664837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:40711)
0928 02:44:09.664969 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.665015 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260928 02:44:09.665071 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.664837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:40489)
0928 02:44:09.664962 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.665020 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260928 02:44:09.665112 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:09.664928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:41549)
0928 02:44:09.665038 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:09.665069 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260928 02:44:10.665778 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.665473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:44999)
0928 02:44:10.665660 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.665723 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":60,"threads_started":1}
I20260928 02:44:10.665817 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.665578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:45985)
0928 02:44:10.665724 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.665761 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":54,"threads_started":1}
I20260928 02:44:10.665989 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:10.665764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:46371)
0928 02:44:10.665921 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:10.665951 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":58,"threads_started":1}
I20260928 02:44:11.666532 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.666198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:55333)
0928 02:44:11.666404 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.666450 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260928 02:44:11.666606 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.666367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:58437)
0928 02:44:11.666513 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.666552 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260928 02:44:11.666733 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:11.666548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:36539)
0928 02:44:11.666668 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:11.666697 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":42,"threads_started":1}
I20260928 02:44:12.667318 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.667009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:55091)
0928 02:44:12.667220 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.667260 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260928 02:44:12.667330 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.667134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:52169)
0928 02:44:12.667239 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.667278 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260928 02:44:12.667443 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:12.667138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:58301)
0928 02:44:12.667373 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:12.667404 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":50,"threads_started":1}
I20260928 02:44:13.668064 3095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.667798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:52491)
0928 02:44:13.667960 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.668006 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260928 02:44:13.668175 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.667815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:55919)
0928 02:44:13.668105 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.668139 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":64,"threads_started":1}
I20260928 02:44:13.668181 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:13.667916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:40453)
0928 02:44:13.668092 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:13.668131 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":54,"threads_started":1}
I20260928 02:44:14.668808 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.668591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:55233)
0928 02:44:14.668711 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.668749 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260928 02:44:14.668972 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.668784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:47015)
0928 02:44:14.668888 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.668924 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260928 02:44:14.669020 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:14.668824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:48335)
0928 02:44:14.668954 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:14.668981 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":44,"threads_started":1}
I20260928 02:44:15.669809 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.669421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:36547)
0928 02:44:15.669745 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.669774 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":51,"threads_started":1}
I20260928 02:44:15.669809 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.669579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:48267)
0928 02:44:15.669727 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.669764 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260928 02:44:15.669826 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.669354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:34123)
0928 02:44:15.669584 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.669627 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":61,"threads_started":1}
I20260928 02:44:15.804245 3104 raft_consensus.cc:482] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b8289534b4fb4ccaa0acb53188440b96)
I20260928 02:44:15.804374 3104 raft_consensus.cc:504] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b8289534b4fb4ccaa0acb53188440b96" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46325 } } peers { permanent_uuid: "addd0cc386854113bc86a10a11131f99" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 38843 } attrs { promote: false } } peers { permanent_uuid: "c47f0d0bda0a4bce94e2c6d494fef06f" member_type: NON_VOTER last_known_addr { host: "127.27.109.188" port: 46045 } attrs { promote: true } }
I20260928 02:44:15.804562 3104 leader_election.cc:288] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325)
I20260928 02:44:15.804955 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.804751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:54228)
0928 02:44:15.804881 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.804916 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260928 02:44:15.805042 2287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:44:15.805423 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:15.805326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.0.0.1:54230)
0928 02:44:15.805378 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:15.805405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 02:44:15.805505 2287 leader_election.cc:334] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b8289534b4fb4ccaa0acb53188440b96 (127.27.109.190:46325): Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
I20260928 02:44:15.805560 2287 leader_election.cc:302] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: addd0cc386854113bc86a10a11131f99; no voters: b8289534b4fb4ccaa0acb53188440b96
I20260928 02:44:15.805661 3104 raft_consensus.cc:2726] T 00000000000000000000000000000000 P addd0cc386854113bc86a10a11131f99 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:44:16.670483 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.670237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.130:33333)
0928 02:44:16.670360 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.670401 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 02:44:16.670712 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.670395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.129:39321)
0928 02:44:16.670647 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.670674 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":52,"threads_started":1}
I20260928 02:44:16.670759 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:16.670448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46325 (local address 127.27.109.131:38645)
0928 02:44:16.670630 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:16.670668 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:46325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.27.109.190:46325,127.27.109.189:38843,127.27.109.188:46045, num_attempts: 1)
I20260928 02:44:16.844623 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2450
I20260928 02:44:16.849721 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2678
I20260928 02:44:16.854058 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2814
I20260928 02:44:16.859220 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2169
I20260928 02:44:16.863411 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 2357
2026-09-28T02:44:16Z chronyd exiting
I20260928 02:44:16.879347 28086 test_util.cc:175] -----------------------------------------------
I20260928 02:44:16.879397 28086 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563399140758-28086-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32981 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:16 dist-test-slave-b5p5 krb5kdc[3112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:16 dist-test-slave-b5p5 krb5kdc[3112](info): set up 2 sockets
Sep 28 02:44:16 dist-test-slave-b5p5 krb5kdc[3112](info): commencing operation
krb5kdc: starting...
W20260928 02:44:17.915052 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:44:17 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab.
Entry for principal oryx/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab.
I20260928 02:44:18.029022 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40835
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab
--principal=oryx/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:18.108186 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:18.108376 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:18.108398 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:18.109457 3125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:18.109489 3125 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:44:18.109503 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:18.109516 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:18.109529 3125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:18.109540 3125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:18.111009 3125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40835
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.190.keytab
--principal=oryx/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:18.111202 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:18.111359 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:18.116008 3131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.116019 3133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.116065 3125 server_base.cc:1034] running on GCE node
W20260928 02:44:18.116019 3130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.116492 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:18.116730 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:18.117866 3125 hybrid_clock.cc:648] HybridClock initialized: now 1790563458117846 us; error 37 us; skew 500 ppm
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:18.125592 3125 init.cc:377] Logged in from keytab as oryx/127.27.109.190@KRBTEST.COM (short username oryx)
I20260928 02:44:18.126068 3125 webserver.cc:460] Webserver started at http://127.27.109.190:37461/ using document root <none> and password file <none>
I20260928 02:44:18.126291 3125 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:18.126367 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:18.126544 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:18.127573 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "0ac4cd6403cc453eb1a07d5b718eab7f"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "346bfe8b85fa0271f13bf966f50aa89f"
server_key_iv: "d55f6d946ef7f808735b39fb105e52df"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.127941 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "0ac4cd6403cc453eb1a07d5b718eab7f"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "346bfe8b85fa0271f13bf966f50aa89f"
server_key_iv: "d55f6d946ef7f808735b39fb105e52df"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.129271 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:44:18.129977 3140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.130143 3125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:18.130225 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "0ac4cd6403cc453eb1a07d5b718eab7f"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "346bfe8b85fa0271f13bf966f50aa89f"
server_key_iv: "d55f6d946ef7f808735b39fb105e52df"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.130308 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:18.174335 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:18.176544 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:18.176707 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:18.181337 3125 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:40835
I20260928 02:44:18.181432 3202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:40835 every 8 connection(s)
I20260928 02:44:18.181740 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:18.182621 3203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.183614 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3125
I20260928 02:44:18.183706 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:44:18.183904 28086 external_mini_cluster.cc:1428] Setting key 1e41d4a1afd0285bdb11d34cdf2082b5
I20260928 02:44:18.185462 3203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Bootstrap starting.
I20260928 02:44:18.186123 3203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.186443 3203 log.cc:824] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:18.187176 3203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: No bootstrap required, opened a new log
I20260928 02:44:18.189530 3203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } }
I20260928 02:44:18.189662 3203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:18.189684 3203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac4cd6403cc453eb1a07d5b718eab7f, State: Initialized, Role: FOLLOWER
I20260928 02:44:18.189824 3203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } }
I20260928 02:44:18.189906 3203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:18.189930 3203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:18.190001 3203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:18.190546 3203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } }
I20260928 02:44:18.190665 3203 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac4cd6403cc453eb1a07d5b718eab7f; no voters:
I20260928 02:44:18.190835 3203 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:18.190891 3208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:18.191058 3208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [term 1 LEADER]: Becoming Leader. State: Replica: 0ac4cd6403cc453eb1a07d5b718eab7f, State: Running, Role: LEADER
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:18.191092 3203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:18.191212 3208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } }
I20260928 02:44:18.191558 3209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } } }
I20260928 02:44:18.191548 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac4cd6403cc453eb1a07d5b718eab7f. Latest consensus state: current_term: 1 leader_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac4cd6403cc453eb1a07d5b718eab7f" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40835 } } }
I20260928 02:44:18.191641 3209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:18.191649 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:18.191864 3213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:18.192378 3213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:18.194360 3213 catalog_manager.cc:1347] Generated new cluster ID: cb21c899d9084875a9cc646bdd1b0709
I20260928 02:44:18.194453 3213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:18.195381 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.184427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.27.109.190:40835)
0928 02:44:18.184622 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.184631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.184664 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:44:18.185221 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.185231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.185316 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.185469 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.186092 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.187474 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.188202 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.188826 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.188941 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.191700 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.191705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.191705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.191716 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.193902 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.194246 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.194249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.194250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.194355 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.194596 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.194596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.194597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.194817 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.194970 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.195094 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.195226 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260928 02:44:18.214176 3213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:18.214910 3213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:18.222393 3213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Generated new TSK 0
I20260928 02:44:18.222584 3213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
I20260928 02:44:18.256731 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40835
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:18.338084 3229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:18.338287 3229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:18.338311 3229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:18.339433 3229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:18.339473 3229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:18.339493 3229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:44:18.339545 3229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:18.340958 3229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:18.341192 3229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:18.341413 3229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:18.346230 3234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.346401 3229 server_base.cc:1034] running on GCE node
W20260928 02:44:18.346230 3235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.346262 3237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.346724 3229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:18.346958 3229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:18.348148 3229 hybrid_clock.cc:648] HybridClock initialized: now 1790563458348140 us; error 34 us; skew 500 ppm
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:18.350627 3229 init.cc:377] Logged in from keytab as oryx/127.27.109.129@KRBTEST.COM (short username oryx)
I20260928 02:44:18.351027 3229 webserver.cc:460] Webserver started at http://127.27.109.129:40839/ using document root <none> and password file <none>
I20260928 02:44:18.351244 3229 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:18.351311 3229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:18.351470 3229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:18.352454 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "69fe8b940d4c450582c8aa4b6cb8da9d"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "19295c6480d61cb7fa94f4f8f538d5a6"
server_key_iv: "11c6e51bb5fd6cb83504fa15ad8b0de3"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.352813 3229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "69fe8b940d4c450582c8aa4b6cb8da9d"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "19295c6480d61cb7fa94f4f8f538d5a6"
server_key_iv: "11c6e51bb5fd6cb83504fa15ad8b0de3"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.354061 3229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:18.354806 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.354975 3229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.355072 3229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "69fe8b940d4c450582c8aa4b6cb8da9d"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "19295c6480d61cb7fa94f4f8f538d5a6"
server_key_iv: "11c6e51bb5fd6cb83504fa15ad8b0de3"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.355152 3229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:18.363449 3229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:18.365056 3229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:18.365247 3229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:18.365767 3229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:18.365825 3229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.365856 3229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:18.365888 3229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:18.373245 3229 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:42581
I20260928 02:44:18.373299 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:42581 every 8 connection(s)
I20260928 02:44:18.373627 3229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:18.375257 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.366102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:48895 (local address 127.27.109.190:40835)
0928 02:44:18.366154 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.366156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.366419 (+ 263us) server_negotiation.cc:407] Connection header received
0928 02:44:18.367147 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.367148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.367159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.367258 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.367911 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.368417 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.369404 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.369621 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.369698 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.372238 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.372240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.372240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.372248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.373818 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.374287 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.374289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.374289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.374346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.374624 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.374625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.374625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.374744 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.374818 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.375077 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.375167 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:44:18.375403 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.366090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.129:48895)
0928 02:44:18.366260 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.366308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.367016 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.367338 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.367342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.367638 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.367816 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.367820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.368527 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.368528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.369312 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.369314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.369703 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.369704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.369821 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.370332 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.370343 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.372144 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.373907 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.373913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.373915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.374198 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.374416 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.374418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.374418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.374555 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.374810 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.374816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.374946 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.375043 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.375195 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 02:44:18.375864 3360 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40835
I20260928 02:44:18.375943 3360 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:18.376118 3360 heartbeater.cc:511] Master 127.27.109.190:40835 requested a full tablet report, sending...
I20260928 02:44:18.376596 3156 ts_manager.cc:194] Registered new tserver with Master: 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581)
I20260928 02:44:18.377458 3156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:48895
I20260928 02:44:18.381573 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3229
I20260928 02:44:18.381704 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:18.381891 28086 external_mini_cluster.cc:1428] Setting key 3303764eaafc369dd0beded2df12ff8c
I20260928 02:44:18.382241 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.376893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.129:41455)
0928 02:44:18.377006 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.377016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.377074 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.377196 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.377196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.377341 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.377398 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.377399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.378072 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.378073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.378854 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.378856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.379166 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.379167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.379208 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.379638 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.379642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.379936 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.381224 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.381224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.381225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.381382 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.381577 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.381578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.381578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.381625 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.381889 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.381890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.381932 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.382019 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.382136 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260928 02:44:18.382265 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.376951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:41455 (local address 127.27.109.190:40835)
0928 02:44:18.376988 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.376990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.377056 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:44:18.377132 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.377133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.377141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.377193 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.377459 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.377956 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.378922 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.379097 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.379155 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.379998 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.379999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.379999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.380003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.381154 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.381448 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.381450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.381450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.381507 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.381696 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.381697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.381698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.381826 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.381884 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.382049 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.382179 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
I20260928 02:44:18.404686 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40835
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:18.483302 3369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:18.483520 3369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:18.483543 3369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:18.484597 3369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:18.484642 3369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:18.484663 3369 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:44:18.484678 3369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:18.486096 3369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:18.486276 3369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:18.486445 3369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:18.491214 3375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.491256 3377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.491215 3374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.491668 3369 server_base.cc:1034] running on GCE node
I20260928 02:44:18.491848 3369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:18.492031 3369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:18.493203 3369 hybrid_clock.cc:648] HybridClock initialized: now 1790563458493185 us; error 31 us; skew 500 ppm
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:18.495687 3369 init.cc:377] Logged in from keytab as oryx/127.27.109.130@KRBTEST.COM (short username oryx)
I20260928 02:44:18.496106 3369 webserver.cc:460] Webserver started at http://127.27.109.130:38761/ using document root <none> and password file <none>
I20260928 02:44:18.496359 3369 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:18.496429 3369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:18.496577 3369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:18.497609 3369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "a526d68187be4206be6d86200700c530"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "fbbadd5f7f18d78deb22275a6e1922cf"
server_key_iv: "30e82b1e32006c8501551764d6760ee4"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.497967 3369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "a526d68187be4206be6d86200700c530"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "fbbadd5f7f18d78deb22275a6e1922cf"
server_key_iv: "30e82b1e32006c8501551764d6760ee4"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.499399 3369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:18.500087 3384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.500240 3369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.500341 3369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "a526d68187be4206be6d86200700c530"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "fbbadd5f7f18d78deb22275a6e1922cf"
server_key_iv: "30e82b1e32006c8501551764d6760ee4"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.500414 3369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:18.516191 3369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:18.517954 3369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:18.518121 3369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:18.518735 3369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:18.518797 3369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.518828 3369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:18.518848 3369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.525357 3369 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:38155
I20260928 02:44:18.525480 3499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:38155 every 8 connection(s)
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:18.525777 3369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:18.528656 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.519065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:57811 (local address 127.27.109.190:40835)
0928 02:44:18.519111 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.519113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.519310 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:44:18.519841 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.519843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.519856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.519911 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.520559 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.521356 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.522932 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.523245 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.523340 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.526120 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.526121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.526122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.526130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.527580 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.527929 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.527930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.527930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.527978 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.528198 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.528199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.528199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.528293 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.528342 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.528542 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.528596 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:44:18.528795 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3369
I20260928 02:44:18.528896 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:18.528947 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.519059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.130:57811)
0928 02:44:18.519193 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.519228 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.519722 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.519985 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.519989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.520282 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.520465 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.520470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.521449 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.521450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.522844 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.522846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.523358 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.523359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.523502 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.524177 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.524189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.526007 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.527657 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.527661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.527663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.527862 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.528036 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.528036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.528036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.528126 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.528347 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.528351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.528455 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.528532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.528709 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 02:44:18.529105 28086 external_mini_cluster.cc:1428] Setting key d190f7755532fda7c1080d70443308e5
I20260928 02:44:18.529457 3500 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40835
I20260928 02:44:18.529541 3500 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:18.529735 3500 heartbeater.cc:511] Master 127.27.109.190:40835 requested a full tablet report, sending...
I20260928 02:44:18.531450 3157 ts_manager.cc:194] Registered new tserver with Master: a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:18.531978 3157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:57811
I20260928 02:44:18.536178 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.530549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:48503 (local address 127.27.109.190:40835)
0928 02:44:18.530583 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.530585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.530710 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:44:18.530781 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.530782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.530792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.530836 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.531129 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.531586 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.532470 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.532684 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.532767 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.533701 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.533702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.533703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.533709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.535048 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.535392 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.535393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.535393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.535449 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.535660 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.535661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.535662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.535796 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.535875 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.536026 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.536096 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:44:18.536194 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.530492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.130:48503)
0928 02:44:18.530634 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.530646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.530724 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.530849 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.530850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.531011 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.531055 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.531056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.531683 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.531684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.532391 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.532393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.532769 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.532770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.532833 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.533300 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.533304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.533609 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.535131 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.535132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.535132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.535305 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.535521 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.535522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.535522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.535579 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.535867 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.535868 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.535911 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.536000 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.536094 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
I20260928 02:44:18.554880 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:40835
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:18.635349 3509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:18.635578 3509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:18.635615 3509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:18.636751 3509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:18.636816 3509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:18.636852 3509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:44:18.636893 3509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:18.638353 3509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.190:40835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:18.638656 3509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:18.638901 3509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:18.643620 3515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.643649 3517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:18.643844 3514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:18.643980 3509 server_base.cc:1034] running on GCE node
I20260928 02:44:18.644157 3509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:18.644378 3509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:18.659762 3509 hybrid_clock.cc:648] HybridClock initialized: now 1790563458658100 us; error 1672 us; skew 500 ppm
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:18.662218 3509 init.cc:377] Logged in from keytab as oryx/127.27.109.131@KRBTEST.COM (short username oryx)
I20260928 02:44:18.662735 3509 webserver.cc:460] Webserver started at http://127.27.109.131:39011/ using document root <none> and password file <none>
I20260928 02:44:18.662971 3509 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:18.663039 3509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:18.663196 3509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:18.664252 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "e930f79fc3d942a5a5616227586e2a44"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "ea5c2048329d671253b54e7b8a9cb6c3"
server_key_iv: "137a81e846be6a4b4d1e00462043e45f"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.664615 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "e930f79fc3d942a5a5616227586e2a44"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "ea5c2048329d671253b54e7b8a9cb6c3"
server_key_iv: "137a81e846be6a4b4d1e00462043e45f"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.665966 3509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:18.666733 3524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.666908 3509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:44:18.667006 3509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "e930f79fc3d942a5a5616227586e2a44"
format_stamp: "Formatted at 2026-09-28 02:44:18 on dist-test-slave-b5p5"
server_key: "ea5c2048329d671253b54e7b8a9cb6c3"
server_key_iv: "137a81e846be6a4b4d1e00462043e45f"
server_key_version: "encryptionkey@0"
I20260928 02:44:18.667089 3509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:18.701416 3509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:18.703372 3509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:18.703531 3509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:18.704051 3509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:18.704121 3509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:18.704160 3509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:18.704187 3509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:18.711182 3509 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:37129
I20260928 02:44:18.711277 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:37129 every 8 connection(s)
I20260928 02:44:18.711588 3509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:18.714064 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.704550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:42989 (local address 127.27.109.190:40835)
0928 02:44:18.704605 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.704607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.704803 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:44:18.705368 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.705370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.705382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.705447 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.706280 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.707134 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.708148 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.708493 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.708574 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.711035 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.711037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.711037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.711044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.712636 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.713138 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.713139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.713140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.713195 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.713455 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.713456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.713456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.713567 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.713642 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.713899 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.713975 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:44:18.714313 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.704542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.131:42989)
0928 02:44:18.704687 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.704724 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.705261 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.705525 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.705529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.705883 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.706164 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.706170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.707254 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.707255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.708041 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.708043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.708599 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.708600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.708752 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.709233 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.709242 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.710936 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.712742 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.712749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.712751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.713050 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.713268 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.713269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.713269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.713379 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.713633 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.713638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.713771 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.713879 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.714026 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260928 02:44:18.714802 3640 heartbeater.cc:347] Connected to a master server at 127.27.109.190:40835
I20260928 02:44:18.714880 3640 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:18.715049 3640 heartbeater.cc:511] Master 127.27.109.190:40835 requested a full tablet report, sending...
I20260928 02:44:18.715431 3156 ts_manager.cc:194] Registered new tserver with Master: e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129)
I20260928 02:44:18.715817 3156 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:42989
I20260928 02:44:18.719717 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3509
I20260928 02:44:18.719821 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:18.720048 28086 external_mini_cluster.cc:1428] Setting key c0760a6218b74d38799f6451a0b69ce9
I20260928 02:44:18.720957 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:18.721700 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.715951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.27.109.131:47793)
0928 02:44:18.716082 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.716092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.716153 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.716684 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.716685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.716836 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:18.716881 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.716883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.717489 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.717489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.717979 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.717981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.718274 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.718275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.718316 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.718757 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:18.718762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:18.719026 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:18.720737 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.720738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.720739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.720965 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.721171 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:18.721172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:18.721172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:18.721232 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:18.721475 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:18.721476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:18.721509 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.721559 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.721634 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 02:44:18.721818 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.715949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:47793 (local address 127.27.109.190:40835)
0928 02:44:18.715989 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.715993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.716143 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:44:18.716597 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.716598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.716617 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.716673 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.716970 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.717403 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.718045 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.718212 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.718261 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.719105 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.719107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.719107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.719115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.720647 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.721038 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.721039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.721039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.721099 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.721293 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.721294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.721294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.721410 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.721480 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.721611 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.721688 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:44:18.742141 3206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:18.736919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.27.109.190:40835)
0928 02:44:18.736983 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.736985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.736994 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:44:18.737019 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.737020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.737029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.737100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.737648 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.738123 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.738962 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.739150 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.739237 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.742088 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58062: BlockingRecv error: recv got EOF from 127.0.0.1:58062 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:44:18 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:18.748661 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.742872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.27.109.190:40835)
0928 02:44:18.742911 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.742913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.742981 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:44:18.743023 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.743024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.743031 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.743082 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:18.743493 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.743923 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.744525 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.744687 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.744733 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.746707 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:18.746708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:18.746708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:18.746714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:18.747865 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.748146 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.748147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.748147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.748194 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:18.748335 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:18.748336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:18.748336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:18.748416 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:18.748478 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.748557 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.748608 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:44:18.750020 3156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:18.751673 3156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:18.757830 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.753977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38155 (local address 127.0.0.1:50038)
0928 02:44:18.754291 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.754301 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.754355 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.754608 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.754608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.754610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:18.754648 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.754649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755494 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.755495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.756679 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.756681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757536 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.757614 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.757628 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.757701 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.757750 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":70,"threads_started":1}
I20260928 02:44:18.757860 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.754103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.27.109.130:38155)
0928 02:44:18.754279 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.754284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.754344 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:44:18.754429 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.754431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.754446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.754539 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:18.754724 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755405 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.756755 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757439 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757541 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.757599 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.757714 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.757752 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260928 02:44:18.758067 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.753791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37129 (local address 127.0.0.1:51916)
0928 02:44:18.753945 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.753984 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.754045 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.754442 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.754445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.754446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:18.754547 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.754548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755867 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.755868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.756931 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.756933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757845 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.757910 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.757914 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.757950 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.758000 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":31,"thread_start_us":47,"threads_started":1}
I20260928 02:44:18.758450 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.753917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.27.109.129:42581)
0928 02:44:18.754095 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.754100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.754367 (+ 267us) server_negotiation.cc:407] Connection header received
0928 02:44:18.754514 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.754516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.754532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.754584 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:18.754801 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755844 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.756920 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757679 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757771 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.757830 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.758308 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.758352 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260928 02:44:18.758535 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.754114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.27.109.131:37129)
0928 02:44:18.754288 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.754293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.754305 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:44:18.754343 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.754345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.754361 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.754442 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:18.754672 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755775 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757016 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757738 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.758289 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.758349 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.758372 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.758416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260928 02:44:18.758627 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 1963e6ddd5934445a470c9589bb8d714 (DEFAULT_TABLE table=test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.758870 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.753798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:42581 (local address 127.0.0.1:45888)
0928 02:44:18.754278 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.754289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.754398 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.754671 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.754671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.754674 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:18.754721 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.754722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.755930 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.755930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.756836 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.756838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.757858 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.757858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.758188 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.758193 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.758584 (+ 391us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.758807 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":19,"thread_start_us":67,"threads_started":1}
I20260928 02:44:18.759001 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1963e6ddd5934445a470c9589bb8d714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.759124 3574 tablet_service.cc:1467] Processing CreateTablet for tablet 1963e6ddd5934445a470c9589bb8d714 (DEFAULT_TABLE table=test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.759419 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1963e6ddd5934445a470c9589bb8d714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.759503 3294 tablet_service.cc:1467] Processing CreateTablet for tablet 1963e6ddd5934445a470c9589bb8d714 (DEFAULT_TABLE table=test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.759816 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1963e6ddd5934445a470c9589bb8d714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.762089 3668 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Bootstrap starting.
I20260928 02:44:18.762712 3668 tablet_bootstrap.cc:654] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.762950 3669 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Bootstrap starting.
I20260928 02:44:18.763057 3668 log.cc:824] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:18.763582 3671 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Bootstrap starting.
I20260928 02:44:18.763701 3669 tablet_bootstrap.cc:654] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.763813 3668 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: No bootstrap required, opened a new log
I20260928 02:44:18.763892 3668 ts_tablet_manager.cc:1397] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:18.764060 3669 log.cc:824] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:18.764230 3671 tablet_bootstrap.cc:654] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.764565 3671 log.cc:824] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:18.764881 3669 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: No bootstrap required, opened a new log
I20260928 02:44:18.764948 3669 ts_tablet_manager.cc:1397] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:18.765245 3671 tablet_bootstrap.cc:492] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: No bootstrap required, opened a new log
I20260928 02:44:18.765324 3671 ts_tablet_manager.cc:1397] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:18.765988 3668 raft_consensus.cc:348] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.766148 3668 raft_consensus.cc:374] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:18.766185 3668 raft_consensus.cc:729] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Initialized, Role: FOLLOWER
I20260928 02:44:18.766314 3668 consensus_queue.cc:260] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.766578 3668 ts_tablet_manager.cc:1428] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:44:18.766805 3640 heartbeater.cc:503] Master 127.27.109.190:40835 was elected leader, sending a full tablet report...
I20260928 02:44:18.766973 3669 raft_consensus.cc:348] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.767133 3669 raft_consensus.cc:374] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:18.767179 3669 raft_consensus.cc:729] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526d68187be4206be6d86200700c530, State: Initialized, Role: FOLLOWER
I20260928 02:44:18.767304 3669 consensus_queue.cc:260] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.767441 3671 raft_consensus.cc:348] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.767567 3500 heartbeater.cc:503] Master 127.27.109.190:40835 was elected leader, sending a full tablet report...
I20260928 02:44:18.767603 3671 raft_consensus.cc:374] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:18.767653 3671 raft_consensus.cc:729] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Initialized, Role: FOLLOWER
I20260928 02:44:18.767552 3669 ts_tablet_manager.cc:1428] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:18.767763 3671 consensus_queue.cc:260] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.768006 3360 heartbeater.cc:503] Master 127.27.109.190:40835 was elected leader, sending a full tablet report...
I20260928 02:44:18.768005 3671 ts_tablet_manager.cc:1428] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260928 02:44:18.776255 3501 tablet.cc:2369] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:44:18.874627 3361 tablet.cc:2369] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:44:18.962252 3641 tablet.cc:2369] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:18.969455 3676 raft_consensus.cc:482] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:18.969559 3676 raft_consensus.cc:504] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.969987 3676 leader_election.cc:288] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:18.973042 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.970450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:57167 (local address 127.27.109.130:38155)
0928 02:44:18.970523 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.970525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.970568 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:44:18.970665 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.970666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.970675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.970749 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:18.970903 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.971355 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972341 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.972698 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972782 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.972790 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.972940 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.972974 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:44:18.973090 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.970285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:45233 (local address 127.27.109.131:37129)
0928 02:44:18.970349 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.970352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.970435 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:44:18.970594 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.970596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.970607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.970692 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:18.970820 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.971345 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972379 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.972724 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972822 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.972831 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.972989 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.973023 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:44:18.973114 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.970186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37129 (local address 127.27.109.129:45233)
0928 02:44:18.970342 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.970355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.970454 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.970697 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.970698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.970700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:18.970767 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.970769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.971439 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.971440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.972253 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.972255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.972823 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.972905 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.972908 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.972981 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.973021 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":46,"thread_start_us":76,"threads_started":1}
I20260928 02:44:18.973264 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.970379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38155 (local address 127.27.109.129:57167)
0928 02:44:18.970512 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:18.970521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:18.970583 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:18.970753 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:18.970755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:18.970757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:18.970822 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.970824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.971452 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.971453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.972243 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:18.972246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.972778 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.972779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:18.972846 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.972850 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:18.972932 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:44:18.972971 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":26,"thread_start_us":58,"threads_started":1}
I20260928 02:44:18.973448 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1963e6ddd5934445a470c9589bb8d714" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44" is_pre_election: true
I20260928 02:44:18.973579 3591 raft_consensus.cc:2454] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:18.973773 3248 leader_election.cc:302] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:18.973760 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1963e6ddd5934445a470c9589bb8d714" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530" is_pre_election: true
I20260928 02:44:18.973896 3454 raft_consensus.cc:2454] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:18.973948 3676 raft_consensus.cc:2781] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:18.974022 3676 raft_consensus.cc:482] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:18.974059 3676 raft_consensus.cc:3037] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:18.974836 3676 raft_consensus.cc:504] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.974961 3676 leader_election.cc:288] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Requested vote from peers e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:18.975140 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1963e6ddd5934445a470c9589bb8d714" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530"
I20260928 02:44:18.975128 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1963e6ddd5934445a470c9589bb8d714" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44"
I20260928 02:44:18.975214 3454 raft_consensus.cc:3037] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:18.975236 3591 raft_consensus.cc:3037] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:18.975936 3454 raft_consensus.cc:2454] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:18.976080 3591 raft_consensus.cc:2454] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:18.976083 3248 leader_election.cc:302] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, a526d68187be4206be6d86200700c530; no voters:
I20260928 02:44:18.976197 3676 raft_consensus.cc:2781] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:18.976327 3676 raft_consensus.cc:686] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Becoming Leader. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Running, Role: LEADER
I20260928 02:44:18.976424 3676 consensus_queue.cc:237] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } }
I20260928 02:44:18.977211 3157 catalog_manager.cc:5495] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129). New cstate: current_term: 1 leader_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } health_report { overall_health: UNKNOWN } } }
I20260928 02:44:18.986405 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:18.984326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.27.109.129:42581)
0928 02:44:18.984377 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:18.984379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:18.984385 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:44:18.984424 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:18.984425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:18.984438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:18.984491 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:18.984601 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.985018 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.985747 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:18.985934 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:18.985992 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:18.986135 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:18.986217 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:18.986272 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:18.986307 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:44:18.986321 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:44:18.988013 3454 raft_consensus.cc:1261] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:18.988013 3591 raft_consensus.cc:1261] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:18.988318 3676 consensus_queue.cc:1035] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:18.988476 3679 consensus_queue.cc:1035] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:18.991173 3682 mvcc.cc:205] Tried to move back new op lower bound from 7334147928012505088 to 7334147927967920128. Current Snapshot: MvccSnapshot[applied={T|T < 7334147928012505088 or (T in {7334147928012505088})}]
I20260928 02:44:18.991827 3684 mvcc.cc:205] Tried to move back new op lower bound from 7334147928012505088 to 7334147927967920128. Current Snapshot: MvccSnapshot[applied={T|T < 7334147928012505088}]
I20260928 02:44:18.992753 3156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:18.992849 3156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:18.993526 3156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Sending DeleteTablet for 3 replicas of tablet 1963e6ddd5934445a470c9589bb8d714
I20260928 02:44:18.993806 3294 tablet_service.cc:1514] Processing DeleteTablet for tablet 1963e6ddd5934445a470c9589bb8d714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:18 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45888
I20260928 02:44:18.993839 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet 1963e6ddd5934445a470c9589bb8d714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:18 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:50038
I20260928 02:44:18.993884 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet 1963e6ddd5934445a470c9589bb8d714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:18 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51916
I20260928 02:44:18.994092 3156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:18.994234 3156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:18.994314 3693 tablet_replica.cc:330] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: stopping tablet replica
I20260928 02:44:18.994431 3693 raft_consensus.cc:2229] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:18.994438 3692 tablet_replica.cc:330] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: stopping tablet replica
I20260928 02:44:18.994565 3693 raft_consensus.cc:2258] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:18.994691 3692 raft_consensus.cc:2229] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:18.994823 3691 tablet_replica.cc:330] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: stopping tablet replica
I20260928 02:44:18.994827 3692 raft_consensus.cc:2258] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:18.994913 3691 raft_consensus.cc:2229] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:18.995018 3691 raft_consensus.cc:2258] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:18.995451 3691 ts_tablet_manager.cc:1905] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:18.995622 3693 ts_tablet_manager.cc:1905] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:18.995764 3294 tablet_service.cc:1467] Processing CreateTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 (DEFAULT_TABLE table=test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.995770 3692 ts_tablet_manager.cc:1905] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:18.995884 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f25e942a86459a9ddd08ad66f373f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.996105 3434 tablet_service.cc:1467] Processing CreateTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 (DEFAULT_TABLE table=test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.996109 3574 tablet_service.cc:1467] Processing CreateTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 (DEFAULT_TABLE table=test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:18.996194 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f25e942a86459a9ddd08ad66f373f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.996199 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f25e942a86459a9ddd08ad66f373f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:18.997241 3671 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Bootstrap starting.
I20260928 02:44:18.997503 3669 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Bootstrap starting.
I20260928 02:44:18.997576 3668 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Bootstrap starting.
I20260928 02:44:18.997684 3671 tablet_bootstrap.cc:654] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.997907 3669 tablet_bootstrap.cc:654] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.997984 3668 tablet_bootstrap.cc:654] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:18.998158 3693 ts_tablet_manager.cc:1918] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:18.998236 3693 log.cc:1196] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/1963e6ddd5934445a470c9589bb8d714
I20260928 02:44:18.998543 3693 ts_tablet_manager.cc:1939] T 1963e6ddd5934445a470c9589bb8d714 P e930f79fc3d942a5a5616227586e2a44: Deleting consensus metadata
I20260928 02:44:18.998903 3144 catalog_manager.cc:4841] TS e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129): tablet 1963e6ddd5934445a470c9589bb8d714 (table test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]) successfully deleted
I20260928 02:44:18.999249 3671 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: No bootstrap required, opened a new log
I20260928 02:44:18.999300 3671 ts_tablet_manager.cc:1397] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:18.999401 3691 ts_tablet_manager.cc:1918] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:18.999436 3671 raft_consensus.cc:348] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:18.999475 3691 log.cc:1196] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/1963e6ddd5934445a470c9589bb8d714
I20260928 02:44:18.999512 3671 raft_consensus.cc:374] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:18.999538 3671 raft_consensus.cc:729] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Initialized, Role: FOLLOWER
I20260928 02:44:18.999578 3671 consensus_queue.cc:260] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:18.999663 3671 ts_tablet_manager.cc:1428] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:18.999733 3691 ts_tablet_manager.cc:1939] T 1963e6ddd5934445a470c9589bb8d714 P a526d68187be4206be6d86200700c530: Deleting consensus metadata
I20260928 02:44:18.999680 3692 ts_tablet_manager.cc:1918] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:18.999832 3692 log.cc:1196] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/1963e6ddd5934445a470c9589bb8d714
I20260928 02:44:19.000128 3692 ts_tablet_manager.cc:1939] T 1963e6ddd5934445a470c9589bb8d714 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting consensus metadata
I20260928 02:44:19.000109 3144 catalog_manager.cc:4841] TS a526d68187be4206be6d86200700c530 (127.27.109.130:38155): tablet 1963e6ddd5934445a470c9589bb8d714 (table test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]) successfully deleted
I20260928 02:44:19.000367 3669 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: No bootstrap required, opened a new log
I20260928 02:44:19.000437 3669 ts_tablet_manager.cc:1397] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:44:19.000550 3143 catalog_manager.cc:4841] TS 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581): tablet 1963e6ddd5934445a470c9589bb8d714 (table test-table [id=b0e10c08ec1d4313bf5ff77e07d487f0]) successfully deleted
I20260928 02:44:19.000582 3669 raft_consensus.cc:348] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.000675 3669 raft_consensus.cc:374] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.000705 3669 raft_consensus.cc:729] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526d68187be4206be6d86200700c530, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.000756 3669 consensus_queue.cc:260] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.000826 3668 tablet_bootstrap.cc:492] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: No bootstrap required, opened a new log
I20260928 02:44:19.000834 3669 ts_tablet_manager.cc:1428] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.000873 3668 ts_tablet_manager.cc:1397] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260928 02:44:19.001011 3668 raft_consensus.cc:348] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.001087 3668 raft_consensus.cc:374] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.001127 3668 raft_consensus.cc:729] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.001176 3668 consensus_queue.cc:260] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.001268 3668 ts_tablet_manager.cc:1428] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.197459 3676 raft_consensus.cc:482] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:19.197551 3676 raft_consensus.cc:504] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.197772 3676 leader_election.cc:288] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a526d68187be4206be6d86200700c530 (127.27.109.130:38155), e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129)
I20260928 02:44:19.198050 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f25e942a86459a9ddd08ad66f373f5" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530" is_pre_election: true
I20260928 02:44:19.198081 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f25e942a86459a9ddd08ad66f373f5" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44" is_pre_election: true
I20260928 02:44:19.198163 3591 raft_consensus.cc:2454] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:19.198185 3454 raft_consensus.cc:2454] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:19.198324 3248 leader_election.cc:302] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.198439 3676 raft_consensus.cc:2781] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:19.198510 3676 raft_consensus.cc:482] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:19.198531 3676 raft_consensus.cc:3037] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.199236 3676 raft_consensus.cc:504] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.199334 3676 leader_election.cc:288] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Requested vote from peers a526d68187be4206be6d86200700c530 (127.27.109.130:38155), e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129)
I20260928 02:44:19.199501 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f25e942a86459a9ddd08ad66f373f5" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530"
I20260928 02:44:19.199540 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f25e942a86459a9ddd08ad66f373f5" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44"
I20260928 02:44:19.199599 3591 raft_consensus.cc:3037] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.199602 3454 raft_consensus.cc:3037] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.200170 3591 raft_consensus.cc:2454] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:19.200237 3454 raft_consensus.cc:2454] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:19.200359 3248 leader_election.cc:302] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.200471 3676 raft_consensus.cc:2781] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:19.200541 3676 raft_consensus.cc:686] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Becoming Leader. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Running, Role: LEADER
I20260928 02:44:19.200595 3676 consensus_queue.cc:237] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.201193 3156 catalog_manager.cc:5495] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129). New cstate: current_term: 1 leader_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:44:19 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563458, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.190@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:19.249894 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.243116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40835 (local address 127.0.0.1:58076)
0928 02:44:19.243305 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.243316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.243409 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.243548 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.243549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.243862 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:19.243904 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.243907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.244564 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.244564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.245074 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.245075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.245453 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.245454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.245510 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.246036 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:19.246041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:19.247481 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:19.248937 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.248943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.248945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.249136 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.249316 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:19.249316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:19.249317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:19.249371 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:19.249614 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:19.249619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:19.249658 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.249709 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.249792 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260928 02:44:19.249939 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.243218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.27.109.190:40835)
0928 02:44:19.243284 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.243287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.243389 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:44:19.243469 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.243470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.243480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.243544 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.243996 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.244479 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.245171 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.245353 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.245440 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.247563 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.247565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.247565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.247571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:19.248865 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.249205 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.249206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.249206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.249256 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.249444 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.249445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.249446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.249554 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:19.249625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.249743 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.249838 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":40}
I20260928 02:44:19.251960 3156 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:58076:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:44:19.252156 3156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:19.253674 3294 tablet_service.cc:1467] Processing CreateTablet for tablet 218aea90fd5d4d459ebddaca47e6b34e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24db737ef99841b881412a55cc4fa1ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:44:19.253710 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 218aea90fd5d4d459ebddaca47e6b34e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24db737ef99841b881412a55cc4fa1ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:44:19.253738 3574 tablet_service.cc:1467] Processing CreateTablet for tablet 218aea90fd5d4d459ebddaca47e6b34e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24db737ef99841b881412a55cc4fa1ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:44:19.253839 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218aea90fd5d4d459ebddaca47e6b34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.253839 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218aea90fd5d4d459ebddaca47e6b34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.253911 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218aea90fd5d4d459ebddaca47e6b34e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.255462 3669 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: Bootstrap starting.
I20260928 02:44:19.255406 3671 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: Bootstrap starting.
I20260928 02:44:19.255877 3671 tablet_bootstrap.cc:654] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.256402 3669 tablet_bootstrap.cc:654] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.256459 3671 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: No bootstrap required, opened a new log
I20260928 02:44:19.256503 3671 ts_tablet_manager.cc:1397] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:19.256500 3668 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: Bootstrap starting.
I20260928 02:44:19.256623 3671 raft_consensus.cc:348] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.256697 3671 raft_consensus.cc:374] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.256723 3671 raft_consensus.cc:729] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.256771 3671 consensus_queue.cc:260] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.256862 3671 ts_tablet_manager.cc:1428] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.256903 3668 tablet_bootstrap.cc:654] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.256978 3676 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.257019 3669 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: No bootstrap required, opened a new log
I20260928 02:44:19.257056 3669 ts_tablet_manager.cc:1397] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:44:19.257048 3676 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.257229 3669 raft_consensus.cc:348] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.257303 3669 raft_consensus.cc:374] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.257357 3669 raft_consensus.cc:729] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526d68187be4206be6d86200700c530, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.257411 3669 consensus_queue.cc:260] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.257510 3669 ts_tablet_manager.cc:1428] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.257570 3675 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.257706 3675 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.257959 3668 tablet_bootstrap.cc:492] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: No bootstrap required, opened a new log
I20260928 02:44:19.258013 3668 ts_tablet_manager.cc:1397] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:44:19.258155 3668 raft_consensus.cc:348] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.258216 3668 raft_consensus.cc:374] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.258234 3668 raft_consensus.cc:729] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.258303 3668 consensus_queue.cc:260] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.258390 3668 ts_tablet_manager.cc:1428] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.258446 3674 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.258565 3674 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.328477 3674 raft_consensus.cc:482] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:19.328586 3674 raft_consensus.cc:504] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.329043 3674 leader_election.cc:288] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:19.332085 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.329297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:48187 (local address 127.27.109.130:38155)
0928 02:44:19.329340 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.329342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.329451 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:44:19.329521 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.329522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.329531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.329586 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:19.329748 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.330274 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331373 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.331733 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.331826 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.331980 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.332014 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:44:19.332077 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.329256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:35411 (local address 127.27.109.129:42581)
0928 02:44:19.329315 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.329318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.329426 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:44:19.329492 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.329493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.329506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.329565 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:19.329701 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.330258 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331376 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.331728 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331814 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.331823 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.331970 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.332004 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:44:19.332119 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.329177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:42581 (local address 127.27.109.131:35411)
0928 02:44:19.329341 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.329351 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.329435 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.329570 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.329570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.329572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:19.329626 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.329630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.330356 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.330358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.331272 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.331274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.331814 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.331878 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.331882 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.331953 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.331992 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":50,"thread_start_us":75,"threads_started":1}
I20260928 02:44:19.332134 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.329180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:38155 (local address 127.27.109.131:48187)
0928 02:44:19.329374 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.329384 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.329467 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.329590 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.329591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.329607 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:19.329667 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.329669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.330355 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.330356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.331257 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.331259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.331816 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.331817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.331893 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.331897 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.331974 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.332014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":46,"thread_start_us":68,"threads_started":1}
I20260928 02:44:19.332428 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218aea90fd5d4d459ebddaca47e6b34e" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" is_pre_election: true
I20260928 02:44:19.332597 3314 raft_consensus.cc:2454] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 0.
I20260928 02:44:19.332662 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218aea90fd5d4d459ebddaca47e6b34e" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530" is_pre_election: true
I20260928 02:44:19.332751 3454 raft_consensus.cc:2454] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 0.
I20260928 02:44:19.332772 3527 leader_election.cc:302] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.332906 3674 raft_consensus.cc:2781] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:19.332950 3674 raft_consensus.cc:482] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:19.332971 3674 raft_consensus.cc:3037] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.333655 3674 raft_consensus.cc:504] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.333791 3674 leader_election.cc:288] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 election: Requested vote from peers 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:19.333957 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218aea90fd5d4d459ebddaca47e6b34e" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d"
I20260928 02:44:19.333976 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218aea90fd5d4d459ebddaca47e6b34e" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530"
I20260928 02:44:19.334038 3454 raft_consensus.cc:3037] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.334046 3314 raft_consensus.cc:3037] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.334623 3454 raft_consensus.cc:2454] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 1.
I20260928 02:44:19.334810 3314 raft_consensus.cc:2454] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 1.
I20260928 02:44:19.334795 3528 leader_election.cc:302] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a526d68187be4206be6d86200700c530, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.334909 3674 raft_consensus.cc:2781] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:19.335022 3674 raft_consensus.cc:686] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [term 1 LEADER]: Becoming Leader. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Running, Role: LEADER
I20260928 02:44:19.335112 3674 consensus_queue.cc:237] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.335441 3709 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: TxnStatusTablet state changed. Reason: New leader e930f79fc3d942a5a5616227586e2a44. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.335547 3709 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:44:19.335728 3711 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:44:19.335848 3711 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:44:19.335762 3157 catalog_manager.cc:5495] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to e930f79fc3d942a5a5616227586e2a44 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } health_report { overall_health: HEALTHY } } }
I20260928 02:44:19.352018 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.349314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37129 (local address 127.0.0.1:51922)
0928 02:44:19.349431 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.349441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.349493 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.349650 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.349650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.349652 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:44:19.349694 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.349695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.350579 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.350580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.351129 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.351130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.351581 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.351581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.351620 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.351633 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.351886 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.351892 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.351928 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.351964 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:44:19.352020 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.349374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.27.109.131:37129)
0928 02:44:19.349432 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.349434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.349480 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:44:19.349547 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.349548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.349566 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.349641 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:19.349749 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.350503 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.351202 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.351507 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.351578 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.351716 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.351825 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.351896 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.351934 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.351953 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:44:19.352989 3454 raft_consensus.cc:1261] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Refusing update from remote peer e930f79fc3d942a5a5616227586e2a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.353211 3314 raft_consensus.cc:1261] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Refusing update from remote peer e930f79fc3d942a5a5616227586e2a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.353272 3709 consensus_queue.cc:1035] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.353463 3674 consensus_queue.cc:1035] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.354090 3676 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: TxnStatusTablet state changed. Reason: New leader e930f79fc3d942a5a5616227586e2a44. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.354172 3676 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.354408 3675 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: TxnStatusTablet state changed. Reason: New leader e930f79fc3d942a5a5616227586e2a44. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.354522 3675 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.354610 3674 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.354636 3675 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.354672 3674 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:44:19.354664 3683 mvcc.cc:205] Tried to move back new op lower bound from 7334147929508163584 to 7334147929437159424. Current Snapshot: MvccSnapshot[applied={T|T < 7334147929508163584}]
I20260928 02:44:19.354682 3675 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P a526d68187be4206be6d86200700c530: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.356108 3709 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.356184 3709 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P e930f79fc3d942a5a5616227586e2a44: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:44:19.356474 3676 tablet_replica.cc:439] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } } }
I20260928 02:44:19.356563 3676 tablet_replica.cc:442] T 218aea90fd5d4d459ebddaca47e6b34e P 69fe8b940d4c450582c8aa4b6cb8da9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:44:19.360890 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.358297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37129 (local address 127.27.109.129:51923)
0928 02:44:19.358435 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.358446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.358525 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.358680 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.358681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.358683 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:44:19.358723 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.358725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.359568 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.359569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.360117 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.360119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.360560 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.360561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.360598 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.360609 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.360761 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.360762 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.360807 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.360840 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260928 02:44:19.360895 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.358371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:51923 (local address 127.27.109.131:37129)
0928 02:44:19.358404 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.358406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.358499 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:44:19.358585 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.358586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.358594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.358663 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:19.358792 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.359487 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.360190 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.360486 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.360555 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.360678 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.360708 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.360757 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.360814 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.360830 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:44:19.362794 3591 raft_consensus.cc:1261] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.362821 3454 raft_consensus.cc:1261] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.362987 3676 consensus_queue.cc:1035] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.363065 3679 consensus_queue.cc:1035] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.371891 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.368420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:46433 (local address 127.27.109.129:42581)
0928 02:44:19.368459 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.368461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.368560 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:44:19.368606 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.368607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.368615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.368672 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:19.368866 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.369615 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.371288 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.371462 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.371515 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.371718 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.371741 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.371778 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.371830 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.371846 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:44:19.371958 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.368358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:42581 (local address 127.27.109.131:46433)
0928 02:44:19.368489 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:19.368500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:19.368558 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:19.368738 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:19.368739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:19.368741 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:44:19.368792 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.368793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.369708 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.369708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.371202 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:19.371205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.371558 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.371559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:19.371628 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.371642 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.371780 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.371781 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:44:19.371829 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:44:19.371867 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260928 02:44:19.379976 3157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:19.380081 3157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:19.380728 3157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Sending DeleteTablet for 3 replicas of tablet e3f25e942a86459a9ddd08ad66f373f5
I20260928 02:44:19.380982 3294 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45888
I20260928 02:44:19.380988 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51916
I20260928 02:44:19.381018 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f25e942a86459a9ddd08ad66f373f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:50038
I20260928 02:44:19.381065 3692 tablet_replica.cc:330] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: stopping tablet replica
I20260928 02:44:19.381072 3693 tablet_replica.cc:330] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: stopping tablet replica
I20260928 02:44:19.381110 3692 raft_consensus.cc:2229] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:19.381105 3691 tablet_replica.cc:330] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: stopping tablet replica
I20260928 02:44:19.381126 3693 raft_consensus.cc:2229] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:19.381158 3693 raft_consensus.cc:2258] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.381157 3691 raft_consensus.cc:2229] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:19.381162 3692 raft_consensus.cc:2258] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.381194 3691 raft_consensus.cc:2258] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.381474 3691 ts_tablet_manager.cc:1905] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.381515 3692 ts_tablet_manager.cc:1905] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.383129 3693 ts_tablet_manager.cc:1905] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.383142 3691 ts_tablet_manager.cc:1918] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:44:19.383193 3691 log.cc:1196] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/e3f25e942a86459a9ddd08ad66f373f5
I20260928 02:44:19.383426 3691 ts_tablet_manager.cc:1939] T e3f25e942a86459a9ddd08ad66f373f5 P a526d68187be4206be6d86200700c530: Deleting consensus metadata
I20260928 02:44:19.383726 3144 catalog_manager.cc:4841] TS a526d68187be4206be6d86200700c530 (127.27.109.130:38155): tablet e3f25e942a86459a9ddd08ad66f373f5 (table test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]) successfully deleted
I20260928 02:44:19.384534 3692 ts_tablet_manager.cc:1918] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:44:19.384575 3693 ts_tablet_manager.cc:1918] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:44:19.384598 3692 log.cc:1196] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/e3f25e942a86459a9ddd08ad66f373f5
I20260928 02:44:19.384620 3693 log.cc:1196] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/e3f25e942a86459a9ddd08ad66f373f5
I20260928 02:44:19.384821 3693 ts_tablet_manager.cc:1939] T e3f25e942a86459a9ddd08ad66f373f5 P e930f79fc3d942a5a5616227586e2a44: Deleting consensus metadata
I20260928 02:44:19.384809 3692 ts_tablet_manager.cc:1939] T e3f25e942a86459a9ddd08ad66f373f5 P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting consensus metadata
I20260928 02:44:19.385128 3143 catalog_manager.cc:4841] TS 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581): tablet e3f25e942a86459a9ddd08ad66f373f5 (table test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]) successfully deleted
I20260928 02:44:19.385248 3144 catalog_manager.cc:4841] TS e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129): tablet e3f25e942a86459a9ddd08ad66f373f5 (table test-table [id=7d0bc5f6eae24344b8a9cf7a925f2dc5]) successfully deleted
Sep 28 02:44:19 dist-test-slave-b5p5 krb5kdc[3112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:44:19 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:19 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:44:19.403193 3206 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:19.397304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.27.109.190:40835)
0928 02:44:19.397369 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.397372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.397466 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:44:19.397525 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.397526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.397534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.397577 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.398225 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.398760 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.399830 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.400003 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.400048 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.403115 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58080: BlockingRecv error: recv got EOF from 127.0.0.1:58080 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 02:44:19 dist-test-slave-b5p5 krb5kdc[3112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM
I20260928 02:44:19.412556 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.403972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.27.109.190:40835)
0928 02:44:19.404007 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.404009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.404143 (+ 134us) server_negotiation.cc:407] Connection header received
0928 02:44:19.404221 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.404222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.404230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.404273 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:19.404740 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.405203 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.405883 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.406176 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.406253 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.408918 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:19.408920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:19.408920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:19.408928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:19.410942 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.411740 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.411742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.411743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.411825 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:19.412049 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:19.412050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:19.412051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:19.412162 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:19.412270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.412385 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.412455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:44:19.414028 3157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:19.414201 3157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:19.415532 3574 tablet_service.cc:1467] Processing CreateTablet for tablet b797aa74de96497b872e68324ef61c8d (DEFAULT_TABLE table=test-table [id=15dfbecad7844b4082ce04733cbe7f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.415516 3294 tablet_service.cc:1467] Processing CreateTablet for tablet b797aa74de96497b872e68324ef61c8d (DEFAULT_TABLE table=test-table [id=15dfbecad7844b4082ce04733cbe7f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.415657 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797aa74de96497b872e68324ef61c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.415656 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797aa74de96497b872e68324ef61c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.416656 3671 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Bootstrap starting.
I20260928 02:44:19.416719 3434 tablet_service.cc:1467] Processing CreateTablet for tablet b797aa74de96497b872e68324ef61c8d (DEFAULT_TABLE table=test-table [id=15dfbecad7844b4082ce04733cbe7f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.416832 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797aa74de96497b872e68324ef61c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.416932 3668 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Bootstrap starting.
I20260928 02:44:19.416944 3671 tablet_bootstrap.cc:654] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.417188 3668 tablet_bootstrap.cc:654] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.417732 3668 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: No bootstrap required, opened a new log
I20260928 02:44:19.417786 3668 ts_tablet_manager.cc:1397] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:44:19.417894 3668 raft_consensus.cc:348] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.417969 3668 raft_consensus.cc:374] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.417991 3668 raft_consensus.cc:729] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.418033 3668 consensus_queue.cc:260] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.418129 3668 ts_tablet_manager.cc:1428] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.418216 3669 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Bootstrap starting.
I20260928 02:44:19.418561 3671 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: No bootstrap required, opened a new log
I20260928 02:44:19.418613 3671 ts_tablet_manager.cc:1397] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:19.418691 3669 tablet_bootstrap.cc:654] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.418720 3671 raft_consensus.cc:348] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.418788 3671 raft_consensus.cc:374] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.418816 3671 raft_consensus.cc:729] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.418865 3671 consensus_queue.cc:260] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.418949 3671 ts_tablet_manager.cc:1428] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:19.419842 3669 tablet_bootstrap.cc:492] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: No bootstrap required, opened a new log
I20260928 02:44:19.419899 3669 ts_tablet_manager.cc:1397] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:19.420040 3669 raft_consensus.cc:348] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.420111 3669 raft_consensus.cc:374] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.420133 3669 raft_consensus.cc:729] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526d68187be4206be6d86200700c530, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.420173 3669 consensus_queue.cc:260] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.420275 3669 ts_tablet_manager.cc:1428] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:19.663789 3674 raft_consensus.cc:482] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:19.663894 3674 raft_consensus.cc:504] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.664083 3674 leader_election.cc:288] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:19.664359 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797aa74de96497b872e68324ef61c8d" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530" is_pre_election: true
I20260928 02:44:19.664363 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797aa74de96497b872e68324ef61c8d" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" is_pre_election: true
I20260928 02:44:19.664448 3454 raft_consensus.cc:2454] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 0.
I20260928 02:44:19.664467 3314 raft_consensus.cc:2454] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 0.
I20260928 02:44:19.664621 3528 leader_election.cc:302] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a526d68187be4206be6d86200700c530, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.664703 3674 raft_consensus.cc:2781] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:19.664748 3674 raft_consensus.cc:482] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:19.664763 3674 raft_consensus.cc:3037] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.665412 3674 raft_consensus.cc:504] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.665545 3674 leader_election.cc:288] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 election: Requested vote from peers 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581), a526d68187be4206be6d86200700c530 (127.27.109.130:38155)
I20260928 02:44:19.665673 3314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797aa74de96497b872e68324ef61c8d" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d"
I20260928 02:44:19.665695 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797aa74de96497b872e68324ef61c8d" candidate_uuid: "e930f79fc3d942a5a5616227586e2a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530"
I20260928 02:44:19.665750 3314 raft_consensus.cc:3037] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.665756 3454 raft_consensus.cc:3037] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.666530 3454 raft_consensus.cc:2454] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 1.
I20260928 02:44:19.666577 3314 raft_consensus.cc:2454] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e930f79fc3d942a5a5616227586e2a44 in term 1.
I20260928 02:44:19.666716 3527 leader_election.cc:302] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.666828 3674 raft_consensus.cc:2781] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:19.666899 3674 raft_consensus.cc:686] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 1 LEADER]: Becoming Leader. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Running, Role: LEADER
I20260928 02:44:19.666951 3674 consensus_queue.cc:237] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.667521 3157 catalog_manager.cc:5495] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to e930f79fc3d942a5a5616227586e2a44 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "e930f79fc3d942a5a5616227586e2a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } health_report { overall_health: HEALTHY } } }
I20260928 02:44:19.703191 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.701017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.27.109.131:37129)
0928 02:44:19.701081 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.701083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.701092 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:44:19.701123 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.701124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.701132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.701184 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:19.701303 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.701760 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.702488 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.702659 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.702709 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.702897 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.702921 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.703011 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.703073 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.703089 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:44:19.704107 3454 raft_consensus.cc:1261] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Refusing update from remote peer e930f79fc3d942a5a5616227586e2a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.704144 3314 raft_consensus.cc:1261] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Refusing update from remote peer e930f79fc3d942a5a5616227586e2a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:19.704339 3674 consensus_queue.cc:1035] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.704435 3709 consensus_queue.cc:1035] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:19.707228 3155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:19.707327 3155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:19.707996 3155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Sending DeleteTablet for 3 replicas of tablet b797aa74de96497b872e68324ef61c8d
I20260928 02:44:19.708236 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet b797aa74de96497b872e68324ef61c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:50038
I20260928 02:44:19.708246 3294 tablet_service.cc:1514] Processing DeleteTablet for tablet b797aa74de96497b872e68324ef61c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45888
I20260928 02:44:19.708350 3691 tablet_replica.cc:330] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: stopping tablet replica
I20260928 02:44:19.708415 3691 raft_consensus.cc:2229] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:19.708460 3691 raft_consensus.cc:2258] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.708416 3155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:44:19.708534 3692 tablet_replica.cc:330] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: stopping tablet replica
W20260928 02:44:19.708561 3155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:19.708580 3692 raft_consensus.cc:2229] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:19.708617 3692 raft_consensus.cc:2258] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.708691 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet b797aa74de96497b872e68324ef61c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:19 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51916
I20260928 02:44:19.708770 3693 tablet_replica.cc:330] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: stopping tablet replica
I20260928 02:44:19.708817 3693 raft_consensus.cc:2229] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:19.708902 3693 raft_consensus.cc:2258] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:19.709321 3693 ts_tablet_manager.cc:1905] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.709709 3294 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0ea0b76964fc7823237b157c11f3a (DEFAULT_TABLE table=test-table [id=601272c5f178483eaef8268e9f58754c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.709748 3434 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0ea0b76964fc7823237b157c11f3a (DEFAULT_TABLE table=test-table [id=601272c5f178483eaef8268e9f58754c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.709834 3294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0ea0b76964fc7823237b157c11f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.709847 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0ea0b76964fc7823237b157c11f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.710862 3693 ts_tablet_manager.cc:1918] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:19.710912 3693 log.cc:1196] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/b797aa74de96497b872e68324ef61c8d
I20260928 02:44:19.711127 3693 ts_tablet_manager.cc:1939] T b797aa74de96497b872e68324ef61c8d P e930f79fc3d942a5a5616227586e2a44: Deleting consensus metadata
I20260928 02:44:19.711179 3671 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: Bootstrap starting.
I20260928 02:44:19.711514 3671 tablet_bootstrap.cc:654] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.711632 3692 ts_tablet_manager.cc:1905] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.711671 3691 ts_tablet_manager.cc:1905] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:19.711740 3669 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Bootstrap starting.
I20260928 02:44:19.712095 3669 tablet_bootstrap.cc:654] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.712659 3144 catalog_manager.cc:4841] TS e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129): tablet b797aa74de96497b872e68324ef61c8d (table test-table [id=15dfbecad7844b4082ce04733cbe7f9a]) successfully deleted
I20260928 02:44:19.712950 3574 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0ea0b76964fc7823237b157c11f3a (DEFAULT_TABLE table=test-table [id=601272c5f178483eaef8268e9f58754c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:19.713019 3671 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: No bootstrap required, opened a new log
I20260928 02:44:19.713047 3574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0ea0b76964fc7823237b157c11f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:19.713064 3671 ts_tablet_manager.cc:1397] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:19.713075 3692 ts_tablet_manager.cc:1918] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:19.713117 3692 log.cc:1196] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/b797aa74de96497b872e68324ef61c8d
I20260928 02:44:19.713171 3671 raft_consensus.cc:348] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.713243 3671 raft_consensus.cc:374] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.713268 3671 raft_consensus.cc:729] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.713321 3692 ts_tablet_manager.cc:1939] T b797aa74de96497b872e68324ef61c8d P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting consensus metadata
I20260928 02:44:19.713634 3143 catalog_manager.cc:4841] TS 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581): tablet b797aa74de96497b872e68324ef61c8d (table test-table [id=15dfbecad7844b4082ce04733cbe7f9a]) successfully deleted
I20260928 02:44:19.713330 3671 consensus_queue.cc:260] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.714008 3671 ts_tablet_manager.cc:1428] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:44:19.714180 3668 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Bootstrap starting.
I20260928 02:44:19.714491 3668 tablet_bootstrap.cc:654] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:19.714613 3669 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: No bootstrap required, opened a new log
I20260928 02:44:19.714664 3669 ts_tablet_manager.cc:1397] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:44:19.714807 3669 raft_consensus.cc:348] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.714876 3669 raft_consensus.cc:374] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.714902 3669 raft_consensus.cc:729] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a526d68187be4206be6d86200700c530, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.714946 3669 consensus_queue.cc:260] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.715029 3691 ts_tablet_manager.cc:1918] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:19.715095 3691 log.cc:1196] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/b797aa74de96497b872e68324ef61c8d
I20260928 02:44:19.715303 3691 ts_tablet_manager.cc:1939] T b797aa74de96497b872e68324ef61c8d P a526d68187be4206be6d86200700c530: Deleting consensus metadata
I20260928 02:44:19.715611 3144 catalog_manager.cc:4841] TS a526d68187be4206be6d86200700c530 (127.27.109.130:38155): tablet b797aa74de96497b872e68324ef61c8d (table test-table [id=15dfbecad7844b4082ce04733cbe7f9a]) successfully deleted
I20260928 02:44:19.715034 3669 ts_tablet_manager.cc:1428] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:19.715863 3668 tablet_bootstrap.cc:492] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: No bootstrap required, opened a new log
I20260928 02:44:19.715931 3668 ts_tablet_manager.cc:1397] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:19.716033 3668 raft_consensus.cc:348] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.716120 3668 raft_consensus.cc:374] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:19.716149 3668 raft_consensus.cc:729] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930f79fc3d942a5a5616227586e2a44, State: Initialized, Role: FOLLOWER
I20260928 02:44:19.716192 3668 consensus_queue.cc:260] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.716279 3668 ts_tablet_manager.cc:1428] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:19.945693 3725 raft_consensus.cc:482] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:19.945775 3725 raft_consensus.cc:504] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.945955 3725 leader_election.cc:288] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a526d68187be4206be6d86200700c530 (127.27.109.130:38155), e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129)
I20260928 02:44:19.946194 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0ea0b76964fc7823237b157c11f3a" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530" is_pre_election: true
I20260928 02:44:19.946197 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0ea0b76964fc7823237b157c11f3a" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44" is_pre_election: true
I20260928 02:44:19.946290 3591 raft_consensus.cc:2454] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:19.946290 3454 raft_consensus.cc:2454] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 0.
I20260928 02:44:19.946434 3248 leader_election.cc:302] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, a526d68187be4206be6d86200700c530; no voters:
I20260928 02:44:19.946569 3725 raft_consensus.cc:2781] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:19.946635 3725 raft_consensus.cc:482] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:19.946678 3725 raft_consensus.cc:3037] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.947372 3725 raft_consensus.cc:504] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.947499 3725 leader_election.cc:288] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Requested vote from peers a526d68187be4206be6d86200700c530 (127.27.109.130:38155), e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129)
I20260928 02:44:19.947652 3454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0ea0b76964fc7823237b157c11f3a" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a526d68187be4206be6d86200700c530"
I20260928 02:44:19.947661 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0ea0b76964fc7823237b157c11f3a" candidate_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930f79fc3d942a5a5616227586e2a44"
I20260928 02:44:19.947741 3591 raft_consensus.cc:3037] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.947741 3454 raft_consensus.cc:3037] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:19.948491 3454 raft_consensus.cc:2454] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:19.948491 3591 raft_consensus.cc:2454] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fe8b940d4c450582c8aa4b6cb8da9d in term 1.
I20260928 02:44:19.948668 3248 leader_election.cc:302] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fe8b940d4c450582c8aa4b6cb8da9d, e930f79fc3d942a5a5616227586e2a44; no voters:
I20260928 02:44:19.948778 3725 raft_consensus.cc:2781] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:19.948889 3725 raft_consensus.cc:686] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Becoming Leader. State: Replica: 69fe8b940d4c450582c8aa4b6cb8da9d, State: Running, Role: LEADER
I20260928 02:44:19.948951 3725 consensus_queue.cc:237] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } }
I20260928 02:44:19.949481 3155 catalog_manager.cc:5495] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129). New cstate: current_term: 1 leader_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fe8b940d4c450582c8aa4b6cb8da9d" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 } health_report { overall_health: UNKNOWN } } }
I20260928 02:44:19.999643 3752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:19.997669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.27.109.129:42581)
0928 02:44:19.997784 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:19.997786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:19.997794 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:19.997816 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:19.997817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:19.997834 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:19.997892 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:19.997978 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.998446 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.999128 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:19.999309 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:19.999363 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:19.999475 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:19.999500 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:19.999536 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:19.999568 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:44:19.999585 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260928 02:44:20.001703 3591 raft_consensus.cc:1261] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:20.001724 3454 raft_consensus.cc:1261] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Refusing update from remote peer 69fe8b940d4c450582c8aa4b6cb8da9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:20.001952 3750 consensus_queue.cc:1035] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a526d68187be4206be6d86200700c530" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:20.002079 3725 consensus_queue.cc:1035] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930f79fc3d942a5a5616227586e2a44" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:20.011521 3155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:20.011603 3155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:20.012176 3155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ac4cd6403cc453eb1a07d5b718eab7f: Sending DeleteTablet for 3 replicas of tablet 39f0ea0b76964fc7823237b157c11f3a
I20260928 02:44:20.012401 3294 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f0ea0b76964fc7823237b157c11f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45888
I20260928 02:44:20.012429 3574 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f0ea0b76964fc7823237b157c11f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51916
I20260928 02:44:20.012511 3692 tablet_replica.cc:330] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: stopping tablet replica
I20260928 02:44:20.012495 3434 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f0ea0b76964fc7823237b157c11f3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:20 UTC) from {username='oryx', principal='oryx/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:50038
I20260928 02:44:20.012524 3693 tablet_replica.cc:330] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: stopping tablet replica
I20260928 02:44:20.012553 3692 raft_consensus.cc:2229] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:20.012576 3691 tablet_replica.cc:330] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: stopping tablet replica
I20260928 02:44:20.012589 3693 raft_consensus.cc:2229] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:20.012607 3692 raft_consensus.cc:2258] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:20.012624 3693 raft_consensus.cc:2258] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:20.012632 3691 raft_consensus.cc:2229] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:20.012667 3691 raft_consensus.cc:2258] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:20.012923 3693 ts_tablet_manager.cc:1905] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:20.012990 3692 ts_tablet_manager.cc:1905] T 39f0ea0b76964fc7823237b157c11f3a P 69fe8b940d4c450582c8aa4b6cb8da9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:20.012995 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3229
I20260928 02:44:20.014341 3691 ts_tablet_manager.cc:1905] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:20.014689 3693 ts_tablet_manager.cc:1918] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:44:20.014740 3693 log.cc:1196] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/39f0ea0b76964fc7823237b157c11f3a
I20260928 02:44:20.014955 3693 ts_tablet_manager.cc:1939] T 39f0ea0b76964fc7823237b157c11f3a P e930f79fc3d942a5a5616227586e2a44: Deleting consensus metadata
I20260928 02:44:20.015275 3144 catalog_manager.cc:4841] TS e930f79fc3d942a5a5616227586e2a44 (127.27.109.131:37129): tablet 39f0ea0b76964fc7823237b157c11f3a (table test-table [id=601272c5f178483eaef8268e9f58754c]) successfully deleted
I20260928 02:44:20.015501 3691 ts_tablet_manager.cc:1918] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:44:20.015535 3691 log.cc:1196] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/39f0ea0b76964fc7823237b157c11f3a
I20260928 02:44:20.015688 3691 ts_tablet_manager.cc:1939] T 39f0ea0b76964fc7823237b157c11f3a P a526d68187be4206be6d86200700c530: Deleting consensus metadata
I20260928 02:44:20.017050 3144 catalog_manager.cc:4841] TS a526d68187be4206be6d86200700c530 (127.27.109.130:38155): tablet 39f0ea0b76964fc7823237b157c11f3a (table test-table [id=601272c5f178483eaef8268e9f58754c]) successfully deleted
W20260928 02:44:20.021677 3143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:42581 (error 108)
I20260928 02:44:20.021956 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3369
I20260928 02:44:20.023418 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:20.022006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:42581 (local address 127.0.0.1:45928)
0928 02:44:20.023348 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:20.023388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":36,"threads_started":1}
W20260928 02:44:20.023533 3143 catalog_manager.cc:4584] TS 69fe8b940d4c450582c8aa4b6cb8da9d (127.27.109.129:42581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39f0ea0b76964fc7823237b157c11f3a: Network error: Client connection negotiation failed: client connection to 127.27.109.129:42581: connect: Connection refused (error 111)
I20260928 02:44:20.026921 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3509
I20260928 02:44:20.032238 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3125
2026-09-28T02:44:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3172 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:20 dist-test-slave-b5p5 krb5kdc[3762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:20 dist-test-slave-b5p5 krb5kdc[3762](info): set up 2 sockets
Sep 28 02:44:20 dist-test-slave-b5p5 krb5kdc[3762](info): commencing operation
krb5kdc: starting...
W20260928 02:44:21.062912 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:44:21 dist-test-slave-b5p5 krb5kdc[3762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:44:21.172950 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:43843
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:43843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:21.251588 3775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:21.251786 3775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:21.251809 3775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:21.253026 3775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:21.253068 3775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:21.253082 3775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:21.253095 3775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:21.253145 3775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:21.254616 3775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:43843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:21.254861 3775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:21.255080 3775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:21.259622 3780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:21.259655 3781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:21.259840 3775 server_base.cc:1034] running on GCE node
W20260928 02:44:21.259802 3783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:21.260185 3775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:21.260422 3775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:21.261557 3775 hybrid_clock.cc:648] HybridClock initialized: now 1790563461261545 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:44:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1236 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:21 dist-test-slave-b5p5 krb5kdc[3788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:21 dist-test-slave-b5p5 krb5kdc[3788](info): set up 2 sockets
Sep 28 02:44:21 dist-test-slave-b5p5 krb5kdc[3788](info): commencing operation
krb5kdc: starting...
W20260928 02:44:22.298868 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:44:22.408783 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37507
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:22.485867 3801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:22.486048 3801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:22.486069 3801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:22.487239 3801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:22.487282 3801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:22.487295 3801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:22.487308 3801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:22.487321 3801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:22.488737 3801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:22.488962 3801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:22.489197 3801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:22.493902 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.493929 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.493934 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.494017 3801 server_base.cc:1034] running on GCE node
I20260928 02:44:22.494376 3801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:22.494669 3801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:22.495848 3801 hybrid_clock.cc:648] HybridClock initialized: now 1790563462495837 us; error 39 us; skew 500 ppm
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:22.498202 3801 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:44:22.498690 3801 webserver.cc:460] Webserver started at http://127.27.109.190:36253/ using document root <none> and password file <none>
I20260928 02:44:22.498898 3801 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:22.498961 3801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:22.499109 3801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:22.500095 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "fca79f34746d48dcac61614d04be14e7"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "e11f96eeff9919db169cf7562affac58"
server_key_iv: "d25f78889d3aa6ac471a37036a1684c4"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.500448 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "fca79f34746d48dcac61614d04be14e7"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "e11f96eeff9919db169cf7562affac58"
server_key_iv: "d25f78889d3aa6ac471a37036a1684c4"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.501675 3801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:22.502388 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.502576 3801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:22.502671 3801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "fca79f34746d48dcac61614d04be14e7"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "e11f96eeff9919db169cf7562affac58"
server_key_iv: "d25f78889d3aa6ac471a37036a1684c4"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.502754 3801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:22.516139 3801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:22.518306 3801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:22.518522 3801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:22.522171 3801 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37507
I20260928 02:44:22.522205 3868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37507 every 8 connection(s)
I20260928 02:44:22.522541 3801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:22.523308 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:22.525465 3869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Bootstrap starting.
I20260928 02:44:22.526013 3869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:22.526306 3869 log.cc:824] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:22.526959 3869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: No bootstrap required, opened a new log
I20260928 02:44:22.528486 3869 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:22.528613 3869 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:22.528635 3869 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca79f34746d48dcac61614d04be14e7, State: Initialized, Role: FOLLOWER
I20260928 02:44:22.528797 3869 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:22.528858 3869 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:22.528879 3869 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:22.528939 3869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:22.529423 3869 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:22.529528 3869 leader_election.cc:302] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca79f34746d48dcac61614d04be14e7; no voters:
I20260928 02:44:22.529700 3869 leader_election.cc:288] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:22.529763 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:22.529958 3869 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:22.530141 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 LEADER]: Becoming Leader. State: Replica: fca79f34746d48dcac61614d04be14e7, State: Running, Role: LEADER
I20260928 02:44:22.530228 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:22.530597 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca79f34746d48dcac61614d04be14e7. Latest consensus state: current_term: 1 leader_uuid: "fca79f34746d48dcac61614d04be14e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } } }
I20260928 02:44:22.530609 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca79f34746d48dcac61614d04be14e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } } }
I20260928 02:44:22.530699 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:22.530705 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:22.530908 3881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:22.531448 3881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:22.532609 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3801
I20260928 02:44:22.532716 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:44:22.532909 28086 external_mini_cluster.cc:1428] Setting key cb35bcc4d5b333f13cb6dd7c00d58672
I20260928 02:44:22.533144 3881 catalog_manager.cc:1347] Generated new cluster ID: 07e0f6707c834fc196ee4f2982288ec1
I20260928 02:44:22.533192 3881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:22.542086 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.533360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.27.109.190:37507)
0928 02:44:22.533498 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:22.533507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:22.533613 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:44:22.533770 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:22.533775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:22.533815 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:22.533948 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:22.534576 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.535490 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.536265 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.536671 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.536785 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:22.539024 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:22.539027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:22.539028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:22.539038 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:22.540615 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.540978 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.540981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.540981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.541067 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.541279 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.541280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.541281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.541503 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:22.541674 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:22.541748 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:44:22.541880 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260928 02:44:22.551594 3881 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:22.552093 3881 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:22.554692 3881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Generated new TSK 0
I20260928 02:44:22.554857 3881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:44:22.583288 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:22.662329 3895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:22.662535 3895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:22.662559 3895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:22.663702 3895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:22.663743 3895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:22.663769 3895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:22.665179 3895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:22.665407 3895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:22.665640 3895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:22.670372 3900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.670423 3901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.670537 3895 server_base.cc:1034] running on GCE node
W20260928 02:44:22.670421 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.670835 3895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:22.671080 3895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:22.672222 3895 hybrid_clock.cc:648] HybridClock initialized: now 1790563462672203 us; error 34 us; skew 500 ppm
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:22.674664 3895 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:44:22.675073 3895 webserver.cc:460] Webserver started at http://127.27.109.129:43211/ using document root <none> and password file <none>
I20260928 02:44:22.675325 3895 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:22.675396 3895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:22.675545 3895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:22.676533 3895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "6ea42751d300415999c818cd96c26f4b"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "cc7b2ae8d9d6de5f0ad7605cd964ebc2"
server_key_iv: "3010f9104c0771fc32ac2921f2af048e"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.676898 3895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "6ea42751d300415999c818cd96c26f4b"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "cc7b2ae8d9d6de5f0ad7605cd964ebc2"
server_key_iv: "3010f9104c0771fc32ac2921f2af048e"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.678269 3895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:22.679066 3910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.679250 3895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.679364 3895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6ea42751d300415999c818cd96c26f4b"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "cc7b2ae8d9d6de5f0ad7605cd964ebc2"
server_key_iv: "3010f9104c0771fc32ac2921f2af048e"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.679450 3895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:22.691708 3895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:22.693337 3895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:22.693495 3895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:22.693697 3895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:22.693974 3895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:22.694027 3895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.694067 3895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:22.694095 3895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.699676 3895 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37175
I20260928 02:44:22.699740 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37175 every 8 connection(s)
I20260928 02:44:22.700029 3895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:22.707612 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 3895
I20260928 02:44:22.707737 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:22.707952 28086 external_mini_cluster.cc:1428] Setting key e65100c2f3fcf47520fd4a76f34ec1e8
I20260928 02:44:22.708782 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.700671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:54105 (local address 127.27.109.190:37507)
0928 02:44:22.700735 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:22.700737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:22.700928 (+ 191us) server_negotiation.cc:407] Connection header received
0928 02:44:22.701353 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:22.701354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:22.701365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:22.701419 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:22.702224 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.702789 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.703702 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.703903 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.703951 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:22.705904 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:22.705905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:22.705905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:22.705912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:22.707368 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.707831 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.707832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.707832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.707895 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.708165 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.708166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.708166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.708294 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:22.708354 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:22.708607 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:44:22.708695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:44:22.708936 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.700663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:54105)
0928 02:44:22.700804 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:22.700850 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:22.701274 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:22.701465 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:22.701472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:22.701986 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:22.702154 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:22.702158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.702862 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.702863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:22.703628 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:22.703630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.703969 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.703970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:22.704059 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:22.704449 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:22.704457 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:22.705801 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:22.707447 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:22.707454 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:22.707456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:22.707752 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:22.707960 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:22.707960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:22.707961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:22.708091 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:22.708362 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:22.708367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:22.708468 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:44:22.708570 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:44:22.708711 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260928 02:44:22.709326 4024 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37507
I20260928 02:44:22.709403 4024 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:22.709573 4024 heartbeater.cc:511] Master 127.27.109.190:37507 requested a full tablet report, sending...
I20260928 02:44:22.710098 3833 ts_manager.cc:194] Registered new tserver with Master: 6ea42751d300415999c818cd96c26f4b (127.27.109.129:37175)
I20260928 02:44:22.711024 3833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:54105
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:44:22.731300 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:22.808338 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:22.808518 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:22.808540 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:22.809592 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:22.809624 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:22.809649 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:22.810974 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:22.811146 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:22.811297 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:22.815793 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.815910 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.816031 4029 server_base.cc:1034] running on GCE node
W20260928 02:44:22.815807 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.816289 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:22.816519 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:22.817675 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790563462817658 us; error 28 us; skew 500 ppm
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:22.819957 4029 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:44:22.820363 4029 webserver.cc:460] Webserver started at http://127.27.109.130:37055/ using document root <none> and password file <none>
I20260928 02:44:22.820607 4029 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:22.820683 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:22.820828 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:22.821797 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "1e63a180197f4836968a2ee75b063927"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "91a42052feeda1d398b483ca43a06142"
server_key_iv: "204d976c5f8669d80052c581bd7162d7"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.822142 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "1e63a180197f4836968a2ee75b063927"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "91a42052feeda1d398b483ca43a06142"
server_key_iv: "204d976c5f8669d80052c581bd7162d7"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.823468 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:22.824147 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.824314 4029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:22.824409 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "1e63a180197f4836968a2ee75b063927"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "91a42052feeda1d398b483ca43a06142"
server_key_iv: "204d976c5f8669d80052c581bd7162d7"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.824488 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:22.837396 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:22.839027 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:22.839180 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:22.839392 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:22.839661 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:22.839713 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.839751 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:22.839779 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.845340 4029 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44617
I20260928 02:44:22.845386 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44617 every 8 connection(s)
I20260928 02:44:22.845670 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:22.853536 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.846309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:56651 (local address 127.27.109.190:37507)
0928 02:44:22.846387 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:22.846390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:22.846652 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:44:22.847053 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:22.847054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:22.847063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:22.847112 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:22.847910 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.848395 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.849273 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.849456 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.849503 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:22.851357 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:22.851358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:22.851358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:22.851364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:22.852495 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.852824 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.852825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.852825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.852870 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:22.853069 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:22.853070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:22.853070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:22.853159 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:22.853206 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:22.853416 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:44:22.853475 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:44:22.853688 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.846279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:56651)
0928 02:44:22.846483 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:22.846559 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:22.846977 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:22.847160 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:22.847163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:22.847673 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:22.847842 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:22.847845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.848469 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.848470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:22.849204 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:22.849206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.849524 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.849525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:22.849623 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:22.849995 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:22.850002 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:22.851262 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:22.852560 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:22.852565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:22.852567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:22.852762 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:22.852924 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:22.852925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:22.852925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:22.853018 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:22.853210 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:22.853213 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:22.853315 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:44:22.853391 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:44:22.853492 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260928 02:44:22.854161 4158 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37507
I20260928 02:44:22.854259 4158 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:22.854430 4158 heartbeater.cc:511] Master 127.27.109.190:37507 requested a full tablet report, sending...
I20260928 02:44:22.854820 3833 ts_manager.cc:194] Registered new tserver with Master: 1e63a180197f4836968a2ee75b063927 (127.27.109.130:44617)
I20260928 02:44:22.855218 3833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:56651
I20260928 02:44:22.855361 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4029
I20260928 02:44:22.855472 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:22.855647 28086 external_mini_cluster.cc:1428] Setting key bb8e0a78d4c78bf9b29ea9e0698a4b68
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:44:22.877367 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:22.955844 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:22.956032 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:22.956054 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:22.957160 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:22.957201 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:22.957250 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:22.958703 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:22.958925 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:22.959132 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:22.963790 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.963768 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:22.963801 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:22.963930 4163 server_base.cc:1034] running on GCE node
I20260928 02:44:22.964205 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:22.964449 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:22.965608 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790563462965595 us; error 41 us; skew 500 ppm
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:22.968027 4163 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:44:22.968427 4163 webserver.cc:460] Webserver started at http://127.27.109.131:34317/ using document root <none> and password file <none>
I20260928 02:44:22.968655 4163 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:22.968721 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:22.968870 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:22.969815 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "057e456300414187980fb0e77d549933"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "0f8ef1c2a434992af89ca5a5731fc114"
server_key_iv: "7e0c71239641e9b98a3177855f2c581a"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.970167 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "057e456300414187980fb0e77d549933"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "0f8ef1c2a434992af89ca5a5731fc114"
server_key_iv: "7e0c71239641e9b98a3177855f2c581a"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.971504 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:22.972172 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.972323 4163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:22.972420 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "057e456300414187980fb0e77d549933"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "0f8ef1c2a434992af89ca5a5731fc114"
server_key_iv: "7e0c71239641e9b98a3177855f2c581a"
server_key_version: "encryptionkey@0"
I20260928 02:44:22.972496 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:22.987586 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:22.989776 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:22.989939 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:22.990142 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:22.990427 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:22.990550 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.990605 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:22.990634 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:22.996307 4163 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:37279
I20260928 02:44:22.996347 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:37279 every 8 connection(s)
I20260928 02:44:22.996635 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:23.001466 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4163
I20260928 02:44:23.001597 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:23.001786 28086 external_mini_cluster.cc:1428] Setting key 25a4dbe88e1eb300d2b68f8f5935eb3e
Sep 28 02:44:22 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:23.005751 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.997340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:48959 (local address 127.27.109.190:37507)
0928 02:44:22.997404 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:22.997406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:22.997572 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:44:22.998010 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:22.998011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:22.998020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:22.998076 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:22.998908 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.999386 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.000287 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.000464 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.000512 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.002882 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:23.002884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:23.002884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:23.002892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:23.004309 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:23.004758 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:23.004759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:23.004760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:23.004847 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:23.005105 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:23.005106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:23.005107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:23.005233 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:23.005306 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:23.005584 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:44:23.005663 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:44:23.005934 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:22.997291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:48959)
0928 02:44:22.997456 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:22.997493 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:22.997934 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:22.998123 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:22.998126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:22.998661 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:22.998828 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:22.998832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:22.999459 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:22.999460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.000216 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.000218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.000530 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.000530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.000621 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.000989 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.000996 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.002795 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:23.004412 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:23.004419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:23.004421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:23.004632 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:23.004929 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:23.004929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:23.004930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:23.005036 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:23.005309 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:23.005315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:23.005449 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:44:23.005549 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:44:23.005693 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 02:44:23.006417 4292 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37507
I20260928 02:44:23.006527 4292 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:23.006691 4292 heartbeater.cc:511] Master 127.27.109.190:37507 requested a full tablet report, sending...
I20260928 02:44:23.007009 3833 ts_manager.cc:194] Registered new tserver with Master: 057e456300414187980fb0e77d549933 (127.27.109.131:37279)
I20260928 02:44:23.007318 3833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:48959
I20260928 02:44:23.013468 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
Entry for principal oryx/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab.
WARNING: no policy specified for oryx/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
Entry for principal oryx/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab.
WARNING: no policy specified for oryx/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
Entry for principal oryx/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab.
I20260928 02:44:23.081692 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3895
I20260928 02:44:23.086136 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4029
I20260928 02:44:23.090782 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4163
I20260928 02:44:23.094851 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 3801
I20260928 02:44:23.099641 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37507
--webserver_interface=127.27.109.190
--webserver_port=36253
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:23.178339 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:23.178627 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:23.178654 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:23.179817 4301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:23.179859 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:23.179873 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:23.179885 4301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:23.179898 4301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:23.181387 4301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=36253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:23.181645 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:23.181872 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:23.186517 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.186564 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.186528 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:23.186686 4301 server_base.cc:1034] running on GCE node
I20260928 02:44:23.186972 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:23.187206 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:23.188374 4301 hybrid_clock.cc:648] HybridClock initialized: now 1790563463188362 us; error 26 us; skew 500 ppm
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:23.194240 4301 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:44:23.194726 4301 webserver.cc:460] Webserver started at http://127.27.109.190:36253/ using document root <none> and password file <none>
I20260928 02:44:23.194954 4301 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:23.195034 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:23.196409 4301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:44:23.197041 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.197197 4301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:23.197293 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "fca79f34746d48dcac61614d04be14e7"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "e11f96eeff9919db169cf7562affac58"
server_key_iv: "d25f78889d3aa6ac471a37036a1684c4"
server_key_version: "encryptionkey@0"
I20260928 02:44:23.197642 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:23.217125 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:23.218951 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:23.219110 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:23.223107 4301 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37507
I20260928 02:44:23.223143 4368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37507 every 8 connection(s)
I20260928 02:44:23.223517 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:23.223815 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4301
I20260928 02:44:23.224117 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:37175
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:37507
--webserver_port=43211
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:44:23.225143 4369 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:44:23.226109 4369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Bootstrap starting.
I20260928 02:44:23.233978 4369 log.cc:824] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:23.235602 4369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:44:23.235886 4369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Bootstrap complete.
I20260928 02:44:23.238046 4369 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:23.238257 4369 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca79f34746d48dcac61614d04be14e7, State: Initialized, Role: FOLLOWER
I20260928 02:44:23.238435 4369 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:23.238547 4369 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:23.238612 4369 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:23.238659 4369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:44:23.239632 4369 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:23.239741 4369 leader_election.cc:302] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca79f34746d48dcac61614d04be14e7; no voters:
I20260928 02:44:23.239938 4369 leader_election.cc:288] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:44:23.239971 4373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:44:23.240087 4373 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [term 2 LEADER]: Becoming Leader. State: Replica: fca79f34746d48dcac61614d04be14e7, State: Running, Role: LEADER
I20260928 02:44:23.240196 4369 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:23.240171 4373 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } }
I20260928 02:44:23.240796 4375 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca79f34746d48dcac61614d04be14e7. Latest consensus state: current_term: 2 leader_uuid: "fca79f34746d48dcac61614d04be14e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } } }
I20260928 02:44:23.240883 4375 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:23.241160 4382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:23.241187 4374 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fca79f34746d48dcac61614d04be14e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca79f34746d48dcac61614d04be14e7" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37507 } } }
I20260928 02:44:23.241233 4374 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:23.241633 4382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:23.241747 4382 catalog_manager.cc:1259] Loaded cluster ID: 07e0f6707c834fc196ee4f2982288ec1
I20260928 02:44:23.241987 4382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:23.243140 4382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:23.243304 4382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fca79f34746d48dcac61614d04be14e7: Loaded TSK: 0
I20260928 02:44:23.243523 4382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:44:23.307788 4371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:23.308007 4371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:23.308043 4371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:23.309259 4371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:23.309321 4371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:23.309367 4371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:23.310832 4371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.129.keytab
--principal=oryx/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:37175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=43211
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:23.311075 4371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:23.311293 4371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:23.315932 4395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.316051 4396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.315936 4398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:23.316799 4371 server_base.cc:1034] running on GCE node
I20260928 02:44:23.317013 4371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:23.317256 4371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:23.318416 4371 hybrid_clock.cc:648] HybridClock initialized: now 1790563463318403 us; error 31 us; skew 500 ppm
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563463, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:23.320876 4371 init.cc:377] Logged in from keytab as oryx/127.27.109.129@KRBTEST.COM (short username oryx)
I20260928 02:44:23.321350 4371 webserver.cc:460] Webserver started at http://127.27.109.129:43211/ using document root <none> and password file <none>
I20260928 02:44:23.321592 4371 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:23.321645 4371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:23.323136 4371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:23.323741 4405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.323935 4371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:23.324040 4371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6ea42751d300415999c818cd96c26f4b"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "cc7b2ae8d9d6de5f0ad7605cd964ebc2"
server_key_iv: "3010f9104c0771fc32ac2921f2af048e"
server_key_version: "encryptionkey@0"
I20260928 02:44:23.324447 4371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:23.345095 4371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:23.347566 4371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:23.347723 4371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:23.347965 4371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:23.348364 4371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:23.348419 4371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.348452 4371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:23.348471 4371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.355610 4371 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37175
I20260928 02:44:23.355643 4518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37175 every 8 connection(s)
I20260928 02:44:23.356035 4371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:23.359083 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4371
I20260928 02:44:23.359370 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:44617
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:37507
--webserver_port=37055
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.365751 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.356830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:37401)
0928 02:44:23.356991 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.357031 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.357537 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.357975 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.357978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.358592 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.358763 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.358768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.360408 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.360409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.361639 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.361642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.362212 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.362213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.362349 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.362933 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.362942 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.365566 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
W20260928 02:44:23.365957 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.356894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:37401 (local address 127.27.109.190:37507)
0928 02:44:23.357110 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.357118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.357156 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:44:23.357733 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.357740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.357798 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.357934 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.358878 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.360279 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.361730 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.362105 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.362185 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.365731 (+ 3546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:37401: BlockingRecv error: recv got EOF from 127.27.109.129:37401 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260928 02:44:23.366199 4519 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37507 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.190:37507: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.371840 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.366431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:52425)
0928 02:44:23.366543 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.366553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.366588 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.366741 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.366742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.366881 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.366926 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.366927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.367872 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.367873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.368665 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.368667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.369203 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.369204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.369267 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.369737 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.369742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.371767 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 02:44:23.371939 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.366568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:52425 (local address 127.27.109.190:37507)
0928 02:44:23.366608 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.366610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.366623 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:44:23.366645 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.366646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.366657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.366726 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.367008 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.367783 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.368753 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.369113 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.369192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.371881 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:52425: BlockingRecv error: recv got EOF from 127.27.109.129:52425 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.377460 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.372409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:40957)
0928 02:44:23.372501 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.372512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.372561 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.372670 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.372670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.372803 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.372838 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.372839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.373724 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.373724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.374640 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.374642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.375077 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.375078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.375129 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.375545 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.375550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.377399 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 02:44:23.377632 4519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:44:23.377734 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.372466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:40957 (local address 127.27.109.190:37507)
0928 02:44:23.372503 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.372505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.372548 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:44:23.372607 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.372608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.372617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.372668 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.372905 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.373635 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.374719 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.375004 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.375071 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.377491 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:40957: BlockingRecv error: recv got EOF from 127.27.109.129:40957 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:44:23.448772 4523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:23.448952 4523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:23.448974 4523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:23.450111 4523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:23.450151 4523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:23.450202 4523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:23.451650 4523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.130.keytab
--principal=oryx/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:44617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=37055
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:23.452003 4523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:23.452266 4523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:23.456950 4531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.457047 4528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.457023 4529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:23.457243 4523 server_base.cc:1034] running on GCE node
I20260928 02:44:23.457417 4523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:23.457597 4523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:23.458741 4523 hybrid_clock.cc:648] HybridClock initialized: now 1790563463458728 us; error 26 us; skew 500 ppm
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563463, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:23.461028 4523 init.cc:377] Logged in from keytab as oryx/127.27.109.130@KRBTEST.COM (short username oryx)
I20260928 02:44:23.461441 4523 webserver.cc:460] Webserver started at http://127.27.109.130:37055/ using document root <none> and password file <none>
I20260928 02:44:23.461632 4523 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:23.461680 4523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:23.462930 4523 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:23.463426 4538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.463595 4523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:23.463670 4523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "1e63a180197f4836968a2ee75b063927"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "91a42052feeda1d398b483ca43a06142"
server_key_iv: "204d976c5f8669d80052c581bd7162d7"
server_key_version: "encryptionkey@0"
I20260928 02:44:23.463933 4523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:23.481648 4523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:23.483883 4523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:23.483989 4523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:23.484174 4523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:23.484474 4523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:23.484508 4523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.484529 4523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:23.484577 4523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.490021 4523 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44617
I20260928 02:44:23.490093 4651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44617 every 8 connection(s)
I20260928 02:44:23.490425 4523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:23.495126 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4523
I20260928 02:44:23.495426 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:37279
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:37507
--webserver_port=34317
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.497553 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.491121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:47283)
0928 02:44:23.491324 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.491359 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.491793 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.491981 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.491984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.492490 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.492658 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.492663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.493386 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.493387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.494167 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.494170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.494581 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.494582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.494688 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.495150 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.495158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.497402 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
W20260928 02:44:23.497565 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.491141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:47283 (local address 127.27.109.190:37507)
0928 02:44:23.491201 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.491203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.491439 (+ 236us) server_negotiation.cc:407] Connection header received
0928 02:44:23.491869 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.491870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.491882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.491930 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.492726 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.493286 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.494248 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.494434 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.494539 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.497505 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:47283: BlockingRecv error: recv got EOF from 127.27.109.130:47283 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 02:44:23.497773 4652 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37507 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.190:37507: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.502154 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.498004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:42387)
0928 02:44:23.498047 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.498056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.498102 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.498244 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.498245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.498367 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.498405 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.498406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.499077 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.499078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.499566 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.499567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.499971 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.499972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.500034 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.500492 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.500496 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.502081 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:44:23.502207 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.498053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42387 (local address 127.27.109.190:37507)
0928 02:44:23.498107 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.498109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.498134 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:44:23.498155 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.498156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.498164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.498226 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.498502 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.499002 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.499662 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.499891 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.499972 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.502145 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:42387: BlockingRecv error: recv got EOF from 127.27.109.130:42387 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.506419 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.502631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:56701)
0928 02:44:23.502670 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.502678 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.502709 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.502848 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.502849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.502967 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.502989 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.502990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.503563 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.503564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.504048 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.504049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.504402 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.504403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.504454 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.504860 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.504864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.506355 (+ 1491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:44:23.506498 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.502694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:56701 (local address 127.27.109.190:37507)
0928 02:44:23.502730 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.502732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.502751 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:44:23.502768 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.502769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.502776 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.502843 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.503059 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.503485 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.504130 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.504320 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.504404 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.506423 (+ 2019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:56701: BlockingRecv error: recv got EOF from 127.27.109.130:56701 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:44:23.506539 4652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:44:23.574796 4655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:23.574985 4655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:23.575006 4655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:23.576167 4655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:23.576210 4655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:23.576236 4655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:23.577617 4655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/krb5kdc/oryx_127.27.109.131.keytab
--principal=oryx/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:37279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=34317
--tserver_master_addrs=127.27.109.190:37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:23.577848 4655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:23.578068 4655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:23.582773 4663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:23.582955 4655 server_base.cc:1034] running on GCE node
W20260928 02:44:23.582784 4660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:23.582792 4661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:23.583223 4655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:23.583472 4655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:23.584614 4655 hybrid_clock.cc:648] HybridClock initialized: now 1790563463584601 us; error 29 us; skew 500 ppm
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563463, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:23.587075 4655 init.cc:377] Logged in from keytab as oryx/127.27.109.131@KRBTEST.COM (short username oryx)
I20260928 02:44:23.587519 4655 webserver.cc:460] Webserver started at http://127.27.109.131:34317/ using document root <none> and password file <none>
I20260928 02:44:23.587755 4655 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:23.587836 4655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:23.589085 4655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:44:23.589666 4670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.589865 4655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:44:23.589975 4655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "057e456300414187980fb0e77d549933"
format_stamp: "Formatted at 2026-09-28 02:44:22 on dist-test-slave-b5p5"
server_key: "0f8ef1c2a434992af89ca5a5731fc114"
server_key_iv: "7e0c71239641e9b98a3177855f2c581a"
server_key_version: "encryptionkey@0"
I20260928 02:44:23.590327 4655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:23.615439 4655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:23.617656 4655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:23.617813 4655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:23.618028 4655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:23.618352 4655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:23.618407 4655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.618448 4655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:23.618537 4655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:23.624233 4655 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:37279
I20260928 02:44:23.624277 4783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:37279 every 8 connection(s)
I20260928 02:44:23.624589 4655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.630501 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4655
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:44:23.631870 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.625191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:56533 (local address 127.27.109.190:37507)
0928 02:44:23.625236 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.625238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.625449 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:44:23.625961 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.625962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.625974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.626031 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.626939 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.627434 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.628494 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.628700 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.628800 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.631826 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:56533: BlockingRecv error: recv got EOF from 127.27.109.131:56533 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:44:23.631985 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.625147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:56533)
0928 02:44:23.625304 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.625369 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.625853 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.626098 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.626102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.626656 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.626828 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.626842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.627526 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.627527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.628403 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.628405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.628803 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.628804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.628943 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.629363 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.629371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.631767 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
W20260928 02:44:23.632268 4784 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37507 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.190:37507: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.636446 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.632490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:56823)
0928 02:44:23.632526 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.632534 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.632587 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.632737 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.632738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.632887 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.632932 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.632933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.633521 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.633522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.634161 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.634163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.634498 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.634499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.634574 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.634907 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.634912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.636407 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 02:44:23.636551 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.632534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:56823 (local address 127.27.109.190:37507)
0928 02:44:23.632570 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.632572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.632580 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:23.632644 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.632645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.632654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.632699 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.633003 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.633422 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.634239 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.634411 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.634505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.636502 (+ 1997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:56823: BlockingRecv error: recv got EOF from 127.27.109.131:56823 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:44:23.638334 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.631006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.27.109.190:37507)
0928 02:44:23.631119 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.631121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.631130 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:44:23.631182 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.631182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.631191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.631240 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.632020 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.632630 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.633486 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.633688 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.633735 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.635059 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:23.635063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:23.635063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:23.635071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:23.637010 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:23.637363 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:23.637366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:23.637366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:23.637475 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:23.637648 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:23.637648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:23.637649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:23.637916 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:23.638020 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:23.638149 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:44:23.638228 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:23 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:23.640945 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:23.636665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:36367)
0928 02:44:23.636790 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:23.636804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:23.636838 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:23.636953 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:23.636954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:23.637074 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:23.637103 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.637104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.637843 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.637844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.638568 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:23.638570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.638939 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.638940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:23.638994 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.639366 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:23.639371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:23.640855 (+ 1484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260928 02:44:23.640982 4522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:23.636697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:36367 (local address 127.27.109.190:37507)
0928 02:44:23.636724 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:23.636726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:23.636833 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:44:23.636875 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:23.636875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:23.636885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:23.636945 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:23.637175 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.637773 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.638656 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:23.638881 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:23.638941 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:23.640930 (+ 1989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:36367: BlockingRecv error: recv got EOF from 127.27.109.131:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 02:44:23.641139 4784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:24.382723 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:24.378012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:42839)
0928 02:44:24.378220 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:24.378230 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:24.378284 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:24.378419 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:24.378419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:24.378639 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:24.378690 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.378692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.379332 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.379333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.379860 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.379861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.380318 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.380319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.380370 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.380881 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:24.380886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:24.382654 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
W20260928 02:44:24.382840 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:24.378119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:42839 (local address 127.27.109.190:37507)
0928 02:44:24.378249 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:24.378253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:24.378271 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:44:24.378339 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:24.378339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:24.378350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:24.378399 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:24.378761 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.379264 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.379984 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.380222 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.380289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.382749 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:42839: BlockingRecv error: recv got EOF from 127.27.109.129:42839 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:24.511313 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:24.506922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:53685)
0928 02:44:24.507077 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:24.507093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:24.507154 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:24.507312 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:24.507313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:24.507478 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:24.507527 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.507529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.508133 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.508134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.508671 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.508672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.509017 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.509018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.509055 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.509489 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:24.509493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:24.511243 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260928 02:44:24.511475 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:24.507022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:53685 (local address 127.27.109.190:37507)
0928 02:44:24.507080 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:24.507082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:24.507157 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:44:24.507206 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:24.507206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:24.507219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:24.507293 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:24.507592 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.508072 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.508757 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.508951 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.509011 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.511403 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:53685: BlockingRecv error: recv got EOF from 127.27.109.130:53685 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:24 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:24.649582 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:24.641560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:39651)
0928 02:44:24.641707 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:24.641717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:24.641778 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:24.641946 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:24.641947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:24.642154 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:24.642219 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.642221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.643214 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.643215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.643783 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:24.643785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.646608 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.646609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:24.646682 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.647205 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:24.647211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:24.649519 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
W20260928 02:44:24.649772 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:24.641591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:39651 (local address 127.27.109.190:37507)
0928 02:44:24.641656 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:24.641664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:24.641775 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:44:24.641820 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:24.641820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:24.641832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:24.641912 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:24.642306 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.643138 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.643875 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:24.644160 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:24.646555 (+ 2395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:24.649629 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:39651: BlockingRecv error: recv got EOF from 127.27.109.131:39651 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:25.387713 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:25.383214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:60363)
0928 02:44:25.383362 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:25.383372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:25.383420 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:25.383566 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:25.383566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:25.383718 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:25.383765 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.383767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.384425 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.384426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.384971 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.384973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.385335 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.385337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.385384 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.385820 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:25.385824 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:25.387657 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
W20260928 02:44:25.387806 4795 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:25.383328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:60363 (local address 127.27.109.190:37507)
0928 02:44:25.383450 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:25.383452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:25.383459 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:44:25.383479 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:25.383479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:25.383489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:25.383548 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:25.383830 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.384332 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.385043 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.385248 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.385312 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.387735 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:60363: BlockingRecv error: recv got EOF from 127.27.109.129:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:25.516503 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:25.511771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:60105)
0928 02:44:25.511951 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:25.511964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:25.512040 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:25.512163 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:25.512163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:25.512364 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:25.512408 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.512410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.513093 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.513094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.513627 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.513629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.513993 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.513995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.514049 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.514609 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:25.514614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:25.516409 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
W20260928 02:44:25.516551 4795 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:25.511873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:60105 (local address 127.27.109.190:37507)
0928 02:44:25.511955 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:25.511957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:25.512036 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:44:25.512097 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:25.512098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:25.512109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:25.512159 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:25.512488 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.513010 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.513714 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.513921 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.513997 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.516488 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:60105: BlockingRecv error: recv got EOF from 127.27.109.130:60105 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:25 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:25.654421 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:25.650020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:36015)
0928 02:44:25.650154 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:25.650166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:25.650221 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:25.650436 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:25.650438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:25.650631 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:25.650677 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.650678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.651327 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.651328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.651858 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:25.651859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.652235 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.652236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:25.652300 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.652709 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:25.652713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:25.654324 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260928 02:44:25.654556 4795 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:25.650263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:36015 (local address 127.27.109.190:37507)
0928 02:44:25.650324 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:25.650325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:25.650332 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:44:25.650350 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:25.650351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:25.650361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:25.650427 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:25.650757 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.651246 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.651947 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:25.652132 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:25.652221 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:25.654470 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:36015: BlockingRecv error: recv got EOF from 127.27.109.131:36015 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:26.393054 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:26.388269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:49255)
0928 02:44:26.388459 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:26.388474 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:26.388537 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:26.388688 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:26.388689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:26.388850 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:26.388899 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.388900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.389643 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.389644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.390177 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.390178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.390562 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.390563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.390603 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.391056 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:26.391061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:26.392979 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
W20260928 02:44:26.393136 4799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:26.388378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:49255 (local address 127.27.109.190:37507)
0928 02:44:26.388504 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:26.388506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:26.388519 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:44:26.388597 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:26.388597 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:26.388609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:26.388672 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:26.388976 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.389569 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.390270 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.390489 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.390565 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.393066 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:49255: BlockingRecv error: recv got EOF from 127.27.109.129:49255 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:26.521736 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:26.517108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:34885)
0928 02:44:26.517282 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:26.517295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:26.517367 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:26.517520 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:26.517521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:26.517690 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:26.517733 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.517735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.518392 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.518393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.518997 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.518999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.519373 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.519373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.519436 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.519867 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:26.519871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:26.521669 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
W20260928 02:44:26.521848 4799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:26.517208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34885 (local address 127.27.109.190:37507)
0928 02:44:26.517296 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:26.517298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:26.517347 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:44:26.517447 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:26.517448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:26.517460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:26.517506 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:26.517813 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.518310 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.519092 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.519291 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.519372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.521757 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:34885: BlockingRecv error: recv got EOF from 127.27.109.130:34885 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:26 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:26.659412 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:26.654914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:33707)
0928 02:44:26.655112 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:26.655125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:26.655216 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:26.655339 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:26.655340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:26.655515 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:26.655561 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.655563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.656231 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.656232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.656771 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:26.656772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.657143 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.657144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:26.657207 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.657618 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:26.657622 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:26.659308 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
W20260928 02:44:26.659466 4799 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:26.655021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:33707 (local address 127.27.109.190:37507)
0928 02:44:26.655095 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:26.655097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:26.655199 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:44:26.655270 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:26.655271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:26.655285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:26.655336 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:26.655640 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.656146 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.656856 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:26.657056 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:26.657143 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:26.659387 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:33707: BlockingRecv error: recv got EOF from 127.27.109.131:33707 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:27.398029 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:27.393476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:33635)
0928 02:44:27.393628 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:27.393640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:27.393701 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:27.393852 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:27.393853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:27.394017 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:27.394068 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.394070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.394694 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.394695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.395273 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.395274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.395604 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.395605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.395647 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.396196 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:27.396201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:27.397972 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
W20260928 02:44:27.398137 4803 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:27.393575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33635 (local address 127.27.109.190:37507)
0928 02:44:27.393666 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:27.393668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:27.393681 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:44:27.393765 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:27.393766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:27.393776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:27.393829 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:27.394119 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.394628 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.395345 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.395540 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.395588 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.398055 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:33635: BlockingRecv error: recv got EOF from 127.27.109.129:33635 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":43,"threads_started":1}
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:27.526722 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:27.522141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:35535)
0928 02:44:27.522301 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:27.522312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:27.522412 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:27.522586 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:27.522587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:27.522758 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:27.522802 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.522804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.523456 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.523457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.523985 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.523987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.524366 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.524367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.524433 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.524906 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:27.524911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:27.526666 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260928 02:44:27.526849 4803 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:27.522214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:35535 (local address 127.27.109.190:37507)
0928 02:44:27.522282 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:27.522284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:27.522378 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:44:27.522484 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:27.522485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:27.522507 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:27.522566 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:27.522883 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.523375 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.524068 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.524264 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.524369 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.526769 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:35535: BlockingRecv error: recv got EOF from 127.27.109.130:35535 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:27 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:27.664124 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:27.659867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:46817)
0928 02:44:27.660030 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:27.660042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:27.660108 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:27.660229 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:27.660230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:27.660400 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:27.660453 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.660455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.661073 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.661073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.661597 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:27.661599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.661897 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.661898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:27.661939 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.662352 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:27.662356 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:27.664068 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
W20260928 02:44:27.664254 4803 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:27.659978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:46817 (local address 127.27.109.190:37507)
0928 02:44:27.660035 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:27.660037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:27.660117 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:44:27.660152 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:27.660153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:27.660164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:27.660210 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:27.660516 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.661009 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.661663 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:27.661838 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:27.661886 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:27.664170 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:46817: BlockingRecv error: recv got EOF from 127.27.109.131:46817 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:28.405807 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:28.398613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:57995)
0928 02:44:28.398748 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:28.398758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:28.398807 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:28.398926 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:28.398926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:28.399078 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:28.399125 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.399126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.399762 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.399763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.400330 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.400332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.400661 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.400662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.400744 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.403958 (+ 3214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:28.403963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:28.405749 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260928 02:44:28.405993 4807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:28.398687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:57995 (local address 127.27.109.190:37507)
0928 02:44:28.398804 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:28.398806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:28.398814 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:28.398856 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:28.398856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:28.398866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:28.398919 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:28.399187 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.399692 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.400401 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.400595 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.400650 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.405884 (+ 5234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:57995: BlockingRecv error: recv got EOF from 127.27.109.129:57995 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":53,"threads_started":1}
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:28.531807 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:28.527196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:42035)
0928 02:44:28.527331 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:28.527343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:28.527427 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:28.527534 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:28.527535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:28.527734 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:28.527793 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.527795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.528482 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.528484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.529032 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.529034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.529447 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.529448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.529497 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.529959 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:28.529963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:28.531743 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260928 02:44:28.531922 4807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:28.527301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42035 (local address 127.27.109.190:37507)
0928 02:44:28.527346 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:28.527348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:28.527395 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:44:28.527473 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:28.527474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:28.527484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:28.527530 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:28.527864 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.528395 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.529110 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.529385 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.529440 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.531839 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:42035: BlockingRecv error: recv got EOF from 127.27.109.130:42035 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:28 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:28.669308 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:28.664572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:43457)
0928 02:44:28.664761 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:28.664773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:28.664868 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:28.665021 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:28.665022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:28.665197 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:28.665264 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.665266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.665991 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.665992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.666602 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:28.666604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.666963 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.666964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:28.667022 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.667522 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:28.667526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:28.669241 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":103,"threads_started":1}
W20260928 02:44:28.669425 4807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:28.664664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43457 (local address 127.27.109.190:37507)
0928 02:44:28.664750 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:28.664752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:28.664846 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:44:28.664933 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:28.664934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:28.664954 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:28.665017 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:28.665348 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.665898 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.666683 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:28.666879 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:28.666958 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:28.669327 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:43457: BlockingRecv error: recv got EOF from 127.27.109.131:43457 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:29.412556 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:29.406266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:60467)
0928 02:44:29.406395 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:29.406408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:29.406504 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:29.406729 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:29.406730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:29.406942 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:29.407014 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.407017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.408149 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.408151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.409151 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.409154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.409711 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.409713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.409770 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.410312 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:29.410318 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:29.412465 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260928 02:44:29.412711 4811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:29.406305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:60467 (local address 127.27.109.190:37507)
0928 02:44:29.406433 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:29.406434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:29.406505 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:44:29.406618 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:29.406619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:29.406632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:29.406721 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:29.407091 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.408061 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.409253 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.409612 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.409697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.412582 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:60467: BlockingRecv error: recv got EOF from 127.27.109.129:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:29.536855 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:29.532316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:37057)
0928 02:44:29.532490 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:29.532500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:29.532564 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:29.532687 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:29.532688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:29.532843 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:29.532888 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.532889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.533506 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.533506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.534023 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.534025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.534584 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.534586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.534655 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.535051 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:29.535055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:29.536788 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
W20260928 02:44:29.536921 4811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:29.532439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:37057 (local address 127.27.109.190:37507)
0928 02:44:29.532505 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:29.532507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:29.532566 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:44:29.532609 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:29.532610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:29.532620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:29.532668 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:29.532947 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.533434 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.534124 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.534449 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.534580 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.536875 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:37057: BlockingRecv error: recv got EOF from 127.27.109.130:37057 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:29 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:29.674376 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:29.669739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:38641)
0928 02:44:29.669895 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:29.669908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:29.670001 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:29.670168 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:29.670169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:29.670339 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:29.670385 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.670387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.671139 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.671140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.671688 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:29.671689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.672062 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.672063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:29.672125 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.672528 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:29.672532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:29.674318 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
W20260928 02:44:29.674440 4811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:29.669822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:38641 (local address 127.27.109.190:37507)
0928 02:44:29.669898 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:29.669900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:29.669980 (+ 80us) server_negotiation.cc:407] Connection header received
0928 02:44:29.670075 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:29.670076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:29.670089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:29.670162 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:29.670532 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.671039 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.671773 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:29.671979 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:29.672052 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:29.674393 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:38641: BlockingRecv error: recv got EOF from 127.27.109.131:38641 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:30.417361 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:30.413003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:42979)
0928 02:44:30.413140 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:30.413159 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:30.413207 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:30.413335 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:30.413336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:30.413493 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:30.413542 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.413543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.414204 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.414205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.414764 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.414766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.415088 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.415089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.415128 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.415655 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:30.415660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:30.417301 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260928 02:44:30.417456 4815 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:30.413078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:42979 (local address 127.27.109.190:37507)
0928 02:44:30.413206 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:30.413208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:30.413215 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:44:30.413255 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:30.413256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:30.413266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:30.413315 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:30.413602 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.414128 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.414833 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.415025 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.415096 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.417381 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:42979: BlockingRecv error: recv got EOF from 127.27.109.129:42979 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:30.541671 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:30.537329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:33971)
0928 02:44:30.537459 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:30.537470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:30.537537 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:30.537663 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:30.537664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:30.537819 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:30.537865 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.537866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.538489 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.538489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.539019 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.539020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.539378 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.539379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.539416 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.539838 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:30.539842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:30.541614 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260928 02:44:30.541812 4815 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:30.537403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:33971 (local address 127.27.109.190:37507)
0928 02:44:30.537471 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:30.537473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:30.537537 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:44:30.537582 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:30.537582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:30.537592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:30.537645 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:30.537928 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.538414 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.539087 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.539272 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.539370 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.541738 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:33971: BlockingRecv error: recv got EOF from 127.27.109.130:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:30 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:30.679363 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:30.674868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:52459)
0928 02:44:30.675002 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:30.675021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:30.675091 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:30.675215 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:30.675215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:30.675381 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:30.675428 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.675429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.676062 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.676063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.676612 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:30.676613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.676918 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.676918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:30.676962 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.677470 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:30.677475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:30.679241 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260928 02:44:30.679399 4815 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:30.674933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:52459 (local address 127.27.109.190:37507)
0928 02:44:30.674986 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:30.674988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:30.675076 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:44:30.675144 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:30.675144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:30.675156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:30.675196 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:30.675498 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.675982 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.676684 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:30.676858 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:30.676905 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:30.679323 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:52459: BlockingRecv error: recv got EOF from 127.27.109.131:52459 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:31.422027 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:31.417847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:56955)
0928 02:44:31.417982 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:31.417996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:31.418063 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:31.418212 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:31.418213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:31.418361 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:31.418408 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.418410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.419073 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.419074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.419603 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.419605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.419908 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.419909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.419944 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.420351 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:31.420355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:31.421967 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
W20260928 02:44:31.422127 4819 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:31.417956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:56955 (local address 127.27.109.190:37507)
0928 02:44:31.418103 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:31.418105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:31.418112 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:44:31.418131 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:31.418131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:31.418140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:31.418195 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:31.418515 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.419009 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.419668 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.419849 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.419897 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.422051 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:56955: BlockingRecv error: recv got EOF from 127.27.109.129:56955 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:31.546952 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:31.542105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:49519)
0928 02:44:31.542259 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:31.542270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:31.542324 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:31.542474 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:31.542475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:31.542659 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:31.542704 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.542706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.543642 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.543643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.544238 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.544240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.544570 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.544571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.544613 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.545041 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:31.545045 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:31.546858 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260928 02:44:31.546983 4819 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:31.542161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:49519 (local address 127.27.109.190:37507)
0928 02:44:31.542220 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:31.542222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:31.542321 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:44:31.542373 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:31.542374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:31.542384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:31.542437 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:31.542766 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.543569 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.544323 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.544495 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.544553 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.546927 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:49519: BlockingRecv error: recv got EOF from 127.27.109.130:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:31 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:31.684437 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:31.679931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:41705)
0928 02:44:31.680072 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:31.680083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:31.680138 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:31.680258 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:31.680259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:31.680414 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:31.680462 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.680463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.681070 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.681071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.681576 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:31.681577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.681878 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.681878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:31.681917 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.682344 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:31.682348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:31.684345 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260928 02:44:31.684479 4819 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:31.680014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41705 (local address 127.27.109.190:37507)
0928 02:44:31.680081 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:31.680084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:31.680139 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:44:31.680178 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:31.680179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:31.680191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:31.680238 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:31.680530 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.681004 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.681643 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:31.681816 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:31.681864 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:31.684418 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:41705: BlockingRecv error: recv got EOF from 127.27.109.131:41705 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:32.427233 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:32.422512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:51203)
0928 02:44:32.422686 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:32.422701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:32.422769 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:32.422959 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:32.422960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:32.423118 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:32.423165 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.423166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.423771 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.423771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.424283 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.424284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.424608 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.424609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.424645 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.425142 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:32.425147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:32.427177 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
W20260928 02:44:32.427368 4823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:32.422727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:51203 (local address 127.27.109.190:37507)
0928 02:44:32.422846 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:32.422847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:32.422854 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:44:32.422874 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:32.422875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:32.422884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:32.422942 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:32.423228 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.423711 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.424355 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.424545 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.424599 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.427260 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:51203: BlockingRecv error: recv got EOF from 127.27.109.129:51203 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:32.551756 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:32.547433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:34585)
0928 02:44:32.547581 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:32.547599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:32.547656 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:32.547781 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:32.547782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:32.547948 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:32.548007 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.548009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.548620 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.548621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.549131 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.549133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.549439 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.549440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.549481 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.549905 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:32.549909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:32.551698 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
W20260928 02:44:32.551895 4823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:32.547536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34585 (local address 127.27.109.190:37507)
0928 02:44:32.547595 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:32.547597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:32.547643 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:44:32.547709 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:32.547710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:32.547720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:32.547762 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:32.548073 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.548557 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.549197 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.549377 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.549427 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.551815 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:34585: BlockingRecv error: recv got EOF from 127.27.109.130:34585 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:32 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:32.689399 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:32.684869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:43283)
0928 02:44:32.685028 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:32.685041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:32.685112 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:32.685259 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:32.685260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:32.685449 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:32.685495 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.685497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.686161 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.686162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.686770 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:32.686772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.687125 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.687126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:32.687190 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.687621 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:32.687626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:32.689305 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260928 02:44:32.689438 4823 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:32.684953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43283 (local address 127.27.109.190:37507)
0928 02:44:32.685033 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:32.685035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:32.685108 (+ 73us) server_negotiation.cc:407] Connection header received
0928 02:44:32.685167 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:32.685168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:32.685182 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:32.685251 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:32.685587 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.686078 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.686862 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:32.687055 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:32.687128 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:32.689379 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:43283: BlockingRecv error: recv got EOF from 127.27.109.131:43283 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:33.434545 4826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:33.427646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:49195)
0928 02:44:33.427782 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:33.427792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:33.427839 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:33.427989 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:33.427990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:33.428141 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:33.428187 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.428188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.428900 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.428901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.429428 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.429430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.429786 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.429787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.429829 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.432682 (+ 2853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:33.432687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:33.434427 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
W20260928 02:44:33.434588 4827 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:33.427738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:49195 (local address 127.27.109.190:37507)
0928 02:44:33.427861 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:33.427862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:33.427870 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:33.427890 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:33.427890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:33.427908 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:33.427974 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:33.428273 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.428792 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.429521 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.429723 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.429776 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.434520 (+ 4744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:49195: BlockingRecv error: recv got EOF from 127.27.109.129:49195 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:33.557124 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:33.552193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:41489)
0928 02:44:33.552392 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:33.552403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:33.552493 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:33.552674 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:33.552676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:33.552860 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:33.552904 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.552906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.553695 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.553696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.554281 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.554283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.554727 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.554728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.554786 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.555283 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:33.555288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:33.557029 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
W20260928 02:44:33.557179 4827 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:33.552296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:41489 (local address 127.27.109.190:37507)
0928 02:44:33.552369 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:33.552371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:33.552475 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:44:33.552579 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:33.552580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:33.552593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:33.552666 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:33.552985 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.553629 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.554367 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.554644 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.554727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.557116 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:41489: BlockingRecv error: recv got EOF from 127.27.109.130:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:33 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:33.694451 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:33.689924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:52963)
0928 02:44:33.690119 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:33.690130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:33.690189 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:33.690311 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:33.690312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:33.690505 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:33.690552 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.690554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.691242 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.691243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.691761 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:33.691762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.692067 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.692068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:33.692107 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.692597 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:33.692601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:33.694394 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
W20260928 02:44:33.694591 4827 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:33.690060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:52963 (local address 127.27.109.190:37507)
0928 02:44:33.690122 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:33.690123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:33.690185 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:44:33.690231 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:33.690232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:33.690244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:33.690293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:33.690625 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.691158 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.691828 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:33.692001 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:33.692052 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:33.694511 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:52963: BlockingRecv error: recv got EOF from 127.27.109.131:52963 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:34.439435 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:34.435057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:33305)
0928 02:44:34.435202 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:34.435213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:34.435263 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:34.435398 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:34.435398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:34.435565 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:34.435611 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.435613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.436256 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.436257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.436777 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.436779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.437149 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.437150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.437189 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.437612 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:34.437616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:34.439345 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260928 02:44:34.439486 4831 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:34.435146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33305 (local address 127.27.109.190:37507)
0928 02:44:34.435281 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:34.435283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:34.435291 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:34.435311 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:34.435311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:34.435322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:34.435381 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:34.435674 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.436192 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.436878 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.437079 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.437144 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.439434 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:33305: BlockingRecv error: recv got EOF from 127.27.109.129:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:34.562215 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:34.557663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:40573)
0928 02:44:34.557819 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:34.557830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:34.557889 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:34.558013 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:34.558014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:34.558182 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:34.558229 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.558231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.558876 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.558877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.559392 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.559393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.559702 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.559703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.559741 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.560179 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:34.560183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:34.562145 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
W20260928 02:44:34.562335 4831 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:34.557724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:40573 (local address 127.27.109.190:37507)
0928 02:44:34.557774 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:34.557776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:34.557877 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:44:34.557935 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:34.557936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:34.557947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:34.557994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:34.558299 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.558807 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.559465 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.559643 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.559693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.562237 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:40573: BlockingRecv error: recv got EOF from 127.27.109.130:40573 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:34 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:34.699539 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:34.694915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:58655)
0928 02:44:34.695088 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:34.695099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:34.695162 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:34.695317 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:34.695318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:34.695497 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:34.695548 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.695549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.696194 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.696195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.696761 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:34.696763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.697091 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.697092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:34.697148 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.697682 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:34.697687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:34.699471 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
W20260928 02:44:34.699690 4831 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:34.694978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:58655 (local address 127.27.109.190:37507)
0928 02:44:34.695072 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:34.695074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:34.695163 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:44:34.695205 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:34.695206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:34.695218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:34.695267 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:34.695620 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.696105 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.696836 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:34.697011 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:34.697062 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:34.699598 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:58655: BlockingRecv error: recv got EOF from 127.27.109.131:58655 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:35.444718 4834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:35.439919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:37561)
0928 02:44:35.440174 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:35.440188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:35.440255 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:35.440382 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:35.440383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:35.440562 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:35.440609 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.440611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.441354 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.441355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.441887 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.441889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.442288 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.442289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.442350 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.442927 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:35.442931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:35.444662 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20260928 02:44:35.444844 4835 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:35.439986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:37561 (local address 127.27.109.190:37507)
0928 02:44:35.440085 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:35.440087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:35.440239 (+ 152us) server_negotiation.cc:407] Connection header received
0928 02:44:35.440296 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:35.440296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:35.440307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:35.440365 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:35.440669 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.441293 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.441971 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.442162 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.442216 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.444756 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:37561: BlockingRecv error: recv got EOF from 127.27.109.129:37561 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:35.567106 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:35.562706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:34503)
0928 02:44:35.562850 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:35.562860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:35.562913 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:35.563037 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:35.563037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:35.563198 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:35.563244 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.563245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.563853 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.563853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.564360 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.564361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.564675 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.564676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.564713 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.565133 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:35.565137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:35.567007 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260928 02:44:35.567154 4835 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:35.562788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34503 (local address 127.27.109.190:37507)
0928 02:44:35.562838 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:35.562841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:35.562902 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:44:35.562955 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:35.562955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:35.562967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:35.563030 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:35.563302 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.563785 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.564426 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.564613 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.564662 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.567085 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:34503: BlockingRecv error: recv got EOF from 127.27.109.130:34503 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:35 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:35.704946 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:35.700175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:37027)
0928 02:44:35.700362 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:35.700377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:35.700457 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:35.700640 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:35.700641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:35.700859 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:35.700902 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.700904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.701632 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.701633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.702180 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:35.702182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.702572 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.702573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:35.702632 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.703122 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:35.703127 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:35.704855 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
W20260928 02:44:35.704985 4835 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:35.700255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:37027 (local address 127.27.109.190:37507)
0928 02:44:35.700326 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:35.700328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:35.700436 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:44:35.700520 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:35.700521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:35.700535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:35.700619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:35.700975 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.701552 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.702266 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:35.702459 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:35.702571 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:35.704925 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:37027: BlockingRecv error: recv got EOF from 127.27.109.131:37027 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:36.449717 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:36.445216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:60791)
0928 02:44:36.445352 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:36.445362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:36.445422 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:36.445566 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:36.445567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:36.445723 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:36.445768 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.445770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.446414 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.446415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.447103 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.447107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.447502 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.447503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.447541 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.447997 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:36.448002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:36.449660 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260928 02:44:36.449920 4839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:36.445263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:60791 (local address 127.27.109.190:37507)
0928 02:44:36.445407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:36.445409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:36.445419 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:44:36.445468 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:36.445468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:36.445479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:36.445542 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:36.445846 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.446347 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.447187 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.447408 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.447496 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.449829 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:60791: BlockingRecv error: recv got EOF from 127.27.109.129:60791 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:36.572063 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:36.567512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:42501)
0928 02:44:36.567663 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:36.567676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:36.567738 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:36.567862 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:36.567863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:36.568059 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:36.568104 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.568106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.568774 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.568776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.569315 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.569317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.569714 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.569715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.569776 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.570211 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:36.570216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:36.571957 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260928 02:44:36.572090 4839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:36.567640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42501 (local address 127.27.109.190:37507)
0928 02:44:36.567695 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:36.567697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:36.567722 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:44:36.567795 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:36.567796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:36.567808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:36.567858 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:36.568187 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.568690 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.569404 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.569638 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.569715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.572033 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:42501: BlockingRecv error: recv got EOF from 127.27.109.130:42501 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:36 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:36.709899 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:36.705450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:58397)
0928 02:44:36.705588 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:36.705600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:36.705687 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:36.705832 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:36.705834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:36.705996 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:36.706053 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.706055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.706741 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.706743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.707290 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:36.707291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.707604 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.707605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:36.707647 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.708059 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:36.708064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:36.709842 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260928 02:44:36.710037 4839 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:36.705493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:58397 (local address 127.27.109.190:37507)
0928 02:44:36.705550 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:36.705553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:36.705654 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:44:36.705752 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:36.705753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:36.705766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:36.705812 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:36.706120 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.706664 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.707365 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:36.707541 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:36.707589 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:36.709937 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:58397: BlockingRecv error: recv got EOF from 127.27.109.131:58397 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:37.454531 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:37.450181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:49359)
0928 02:44:37.450327 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:37.450346 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:37.450431 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:37.450616 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:37.450617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:37.450780 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:37.450828 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.450830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.451440 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.451441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.451959 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.451960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.452280 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.452280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.452319 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.452736 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:37.452740 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:37.454415 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260928 02:44:37.454644 4843 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:37.450251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:49359 (local address 127.27.109.190:37507)
0928 02:44:37.450373 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:37.450375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:37.450419 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:44:37.450541 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:37.450542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:37.450552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:37.450600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:37.450888 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.451377 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.452025 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.452218 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.452267 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.454547 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:49359: BlockingRecv error: recv got EOF from 127.27.109.129:49359 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:37.576983 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:37.572645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:51265)
0928 02:44:37.572774 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:37.572788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:37.572842 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:37.572971 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:37.572972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:37.573139 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:37.573185 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.573187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.573797 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.573798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.574323 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.574324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.574648 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.574649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.574693 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.575105 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:37.575110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:37.576914 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
W20260928 02:44:37.577100 4843 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:37.572761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:51265 (local address 127.27.109.190:37507)
0928 02:44:37.572823 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:37.572825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:37.572835 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:44:37.572890 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:37.572891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:37.572901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:37.572952 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:37.573251 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.573733 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.574394 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.574589 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.574639 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.577006 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:51265: BlockingRecv error: recv got EOF from 127.27.109.130:51265 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:37 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.131@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:37.715380 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:37.710346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.131:43299)
0928 02:44:37.710541 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:37.710553 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:37.710641 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:37.710793 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:37.710794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:37.710999 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:37.711045 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.711047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.711736 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.711737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.712289 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:37.712291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.712818 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.712819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:37.712891 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.713355 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:37.713359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:37.715271 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260928 02:44:37.715416 4843 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:37.710427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:43299 (local address 127.27.109.190:37507)
0928 02:44:37.710514 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:37.710516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:37.710621 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:44:37.710696 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:37.710697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:37.710713 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:37.710785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:37.711128 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.711635 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.712377 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:37.712596 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:37.712818 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:37.715351 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.131:43299: BlockingRecv error: recv got EOF from 127.27.109.131:43299 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:44:38 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:38 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.129@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:38.463165 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:38.454989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.129:39359)
0928 02:44:38.455109 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:38.455119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:38.455164 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:38.455318 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:38.455319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:38.455465 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:38.455512 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.455513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.456128 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.456129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.456648 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.456649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.457001 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.457002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.457038 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.461143 (+ 4105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:38.461147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:38.463102 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
W20260928 02:44:38.463305 4847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:38.455043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:39359 (local address 127.27.109.190:37507)
0928 02:44:38.455197 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:38.455198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:38.455206 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:44:38.455227 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:38.455227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:38.455238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:38.455299 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:38.455577 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.456066 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.456722 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.456936 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.456988 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.463216 (+ 6228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.129:39359: BlockingRecv error: recv got EOF from 127.27.109.129:39359 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
Sep 28 02:44:38 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:44:38 dist-test-slave-b5p5 krb5kdc[3788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.130@KRBTEST.COM for oryx/127.27.109.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:44:38.581890 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:38.577469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37507 (local address 127.27.109.130:50801)
0928 02:44:38.577642 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:38.577657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:38.577711 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:38.577834 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:38.577835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:38.577990 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:38.578044 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.578046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.578707 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.578708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.579230 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.579232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.579575 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.579576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.579623 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.580128 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:38.580132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:38.581834 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.190:37507: Server oryx/127.27.109.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
W20260928 02:44:38.582012 4847 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:38.577545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:50801 (local address 127.27.109.190:37507)
0928 02:44:38.577616 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:38.577618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:38.577714 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:44:38.577753 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:38.577753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:38.577763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:38.577815 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:38.578134 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.578640 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.579331 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.579511 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.579562 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.581912 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.130:50801: BlockingRecv error: recv got EOF from 127.27.109.130:50801 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:44:38.639336 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4371
I20260928 02:44:38.643872 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4523
I20260928 02:44:38.647943 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4655
I20260928 02:44:38.651894 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4301
2026-09-28T02:44:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17381 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:38Z Disabled control of system clock
I20260928 02:44:38.678606 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35881
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:44:38.756489 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:38.756680 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:38.756699 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:38.757771 4852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:38.757802 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:38.757813 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:38.757826 4852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:38.757838 4852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:38.759155 4852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:35881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:38.759333 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:38.759538 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:38.764310 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:38.764411 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:38.764320 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:38.764348 4852 server_base.cc:1034] running on GCE node
I20260928 02:44:38.764827 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:38.765093 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:38.766250 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790563478766233 us; error 28 us; skew 500 ppm
I20260928 02:44:38.767519 4852 webserver.cc:460] Webserver started at http://127.27.109.190:42219/ using document root <none> and password file <none>
I20260928 02:44:38.767748 4852 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:38.767815 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:38.767958 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:38.768942 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "c61d5a740fcf8a005ed4918ebe52739d"
server_key_iv: "b598198f06a3e6e174171fdb15a399f6"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.769290 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "c61d5a740fcf8a005ed4918ebe52739d"
server_key_iv: "b598198f06a3e6e174171fdb15a399f6"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.770532 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:38.771248 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:38.771421 4852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:38.771515 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "c61d5a740fcf8a005ed4918ebe52739d"
server_key_iv: "b598198f06a3e6e174171fdb15a399f6"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.771601 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:38.786319 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:38.786681 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:38.786828 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:38.790575 4852 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:35881
I20260928 02:44:38.790599 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:35881 every 8 connection(s)
I20260928 02:44:38.790912 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:38.791499 4919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:38.792215 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4852
I20260928 02:44:38.792325 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:44:38.792505 28086 external_mini_cluster.cc:1428] Setting key ec37705e25e5a02a74febba4947859b7
I20260928 02:44:38.794751 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6: Bootstrap starting.
I20260928 02:44:38.795396 4919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:38.795688 4919 log.cc:824] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:38.796402 4919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6: No bootstrap required, opened a new log
I20260928 02:44:38.798161 4919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } }
I20260928 02:44:38.798329 4919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:38.798372 4919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9814cf568e4d4f5fa0b352a4c5dbeec6, State: Initialized, Role: FOLLOWER
I20260928 02:44:38.798522 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:38.792987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.27.109.190:35881)
0928 02:44:38.793209 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:38.793217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:38.793250 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:44:38.793775 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:38.793783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:38.793859 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:38.794007 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:38.794167 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.795546 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.796331 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.796915 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.797033 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.797615 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:38.797619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:38.797619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:38.797814 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:38.797853 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:38.797866 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:38.797937 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:38.797993 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:44:38.798064 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260928 02:44:38.798524 4919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } }
I20260928 02:44:38.798629 4919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:38.798658 4919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:38.798700 4919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:38.799319 4919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } }
I20260928 02:44:38.799434 4919 leader_election.cc:302] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9814cf568e4d4f5fa0b352a4c5dbeec6; no voters:
I20260928 02:44:38.799578 4919 leader_election.cc:288] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:38.799613 4924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:38.799774 4924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [term 1 LEADER]: Becoming Leader. State: Replica: 9814cf568e4d4f5fa0b352a4c5dbeec6, State: Running, Role: LEADER
I20260928 02:44:38.799836 4919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:38.799914 4924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } }
I20260928 02:44:38.800282 4925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } } }
I20260928 02:44:38.800292 4926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9814cf568e4d4f5fa0b352a4c5dbeec6. Latest consensus state: current_term: 1 leader_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9814cf568e4d4f5fa0b352a4c5dbeec6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35881 } } }
I20260928 02:44:38.800485 4926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:38.800473 4925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:38.800750 4933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:38.801241 4933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:38.802666 4933 catalog_manager.cc:1347] Generated new cluster ID: c916cceaaa6c464c83cce68adef33372
I20260928 02:44:38.802719 4933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:38.805977 4933 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:38.806438 4933 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:38.822176 4933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9814cf568e4d4f5fa0b352a4c5dbeec6: Generated new TSK 0
I20260928 02:44:38.822350 4933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:44:38.828631 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--builtin_ntp_servers=127.27.109.148:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:44:38.907620 4943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:38.907810 4943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:38.907831 4943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:38.908952 4943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:38.908991 4943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:38.909016 4943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:38.910308 4943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:38.910558 4943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:38.910856 4943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:38.915659 4949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:38.915674 4948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:38.915670 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:38.916195 4943 server_base.cc:1034] running on GCE node
I20260928 02:44:38.916379 4943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:38.916602 4943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:38.917780 4943 hybrid_clock.cc:648] HybridClock initialized: now 1790563478917766 us; error 26 us; skew 500 ppm
I20260928 02:44:38.919150 4943 webserver.cc:460] Webserver started at http://127.27.109.129:35791/ using document root <none> and password file <none>
I20260928 02:44:38.919375 4943 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:38.919427 4943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:38.919548 4943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:38.920804 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "e12c3b505f2b4179ad1d8c095f99329e"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "554bc2a5c807e2a8ff432a8af43a7cb8"
server_key_iv: "bcf2bf8f51668243ca5ffd02f3f9d59f"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.921221 4943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "e12c3b505f2b4179ad1d8c095f99329e"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "554bc2a5c807e2a8ff432a8af43a7cb8"
server_key_iv: "bcf2bf8f51668243ca5ffd02f3f9d59f"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.922566 4943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:38.923269 4957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:38.923460 4943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:38.923529 4943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "e12c3b505f2b4179ad1d8c095f99329e"
format_stamp: "Formatted at 2026-09-28 02:44:38 on dist-test-slave-b5p5"
server_key: "554bc2a5c807e2a8ff432a8af43a7cb8"
server_key_iv: "bcf2bf8f51668243ca5ffd02f3f9d59f"
server_key_version: "encryptionkey@0"
I20260928 02:44:38.923637 4943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:38.933914 4943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:38.934181 4943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:38.934324 4943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:38.934589 4943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:38.934894 4943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:38.934930 4943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:38.934952 4943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:38.934963 4943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:38.940435 4943 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:41749
I20260928 02:44:38.940508 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:41749 every 8 connection(s)
I20260928 02:44:38.940819 4943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:38.943123 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 4943
I20260928 02:44:38.943231 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:38.943439 28086 external_mini_cluster.cc:1428] Setting key 7f61e88fe22dc882d56900a0de105692
I20260928 02:44:38.944483 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--builtin_ntp_servers=127.27.109.148:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:44:38.946816 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:38.941529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:57255 (local address 127.27.109.190:35881)
0928 02:44:38.941605 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:38.941607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:38.941834 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:44:38.942364 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:38.942366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:38.942378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:38.942436 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:38.943219 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.943886 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.945324 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.945521 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.945576 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.946561 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:38.946562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:38.946563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:38.946567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:38.946581 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:38.946585 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:38.946641 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:38.946732 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:44:38.946760 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:44:38.947081 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:38.941514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35881 (local address 127.27.109.129:57255)
0928 02:44:38.941716 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:38.941748 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:38.942255 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:38.942524 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:38.942529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:38.942544 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:38.943091 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.943098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.943976 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.943977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.945246 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:38.945248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:38.945610 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:38.945611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:38.945776 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:38.946369 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:38.946369 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:38.946403 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:38.946425 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:38.946643 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:38.946648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:38.946651 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:38.946721 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:44:38.946827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 02:44:38.947594 5071 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35881
I20260928 02:44:38.947680 5071 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:38.947849 5071 heartbeater.cc:511] Master 127.27.109.190:35881 requested a full tablet report, sending...
I20260928 02:44:38.948356 4883 ts_manager.cc:194] Registered new tserver with Master: e12c3b505f2b4179ad1d8c095f99329e (127.27.109.129:41749)
I20260928 02:44:38.948940 4883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.129:57255
W20260928 02:44:39.024075 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.024295 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.024330 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.025480 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.025527 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.025552 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:39.027004 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.027249 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.027596 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.032403 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.032406 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.032387 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.032787 5074 server_base.cc:1034] running on GCE node
I20260928 02:44:39.033006 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.033252 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.034416 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790563479034403 us; error 41 us; skew 500 ppm
I20260928 02:44:39.035584 5074 webserver.cc:460] Webserver started at http://127.27.109.130:40657/ using document root <none> and password file <none>
I20260928 02:44:39.035821 5074 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.035887 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.036026 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.036988 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "0aeb842bf059447f91b2f1ab320ea157"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "a6794c5ed7a1bab4e2858bac64a7e715"
server_key_iv: "9e13adf0f7ae1aca8ecc46de33018dfd"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.037343 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "0aeb842bf059447f91b2f1ab320ea157"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "a6794c5ed7a1bab4e2858bac64a7e715"
server_key_iv: "9e13adf0f7ae1aca8ecc46de33018dfd"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.038616 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:44:39.039322 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.039495 5074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:39.039589 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "0aeb842bf059447f91b2f1ab320ea157"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "a6794c5ed7a1bab4e2858bac64a7e715"
server_key_iv: "9e13adf0f7ae1aca8ecc46de33018dfd"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.039676 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.055573 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.055837 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.055984 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.056190 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:39.056507 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:39.056565 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.056604 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:39.056630 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.062269 5074 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44295
I20260928 02:44:39.062320 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44295 every 8 connection(s)
I20260928 02:44:39.062692 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:39.067873 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.063364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:43665 (local address 127.27.109.190:35881)
0928 02:44:39.063421 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.063423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.063655 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:44:39.064171 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.064173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.064184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.064239 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.064891 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.065388 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.066368 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.066581 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.066669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:39.067561 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.067563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.067563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.067569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.067584 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.067588 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.067673 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.067768 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.067794 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:44:39.068091 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.063352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35881 (local address 127.27.109.130:43665)
0928 02:44:39.063546 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:39.063575 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:39.064065 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:39.064305 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:39.064309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:39.064319 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:39.064799 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:39.064803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.065496 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.065497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:39.066276 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:39.066278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.066676 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.066677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:39.066810 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:39.067369 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:39.067370 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:39.067399 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:39.067422 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:39.067672 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:39.067679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:39.067681 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:44:39.067771 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:44:39.067872 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260928 02:44:39.068630 5202 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35881
I20260928 02:44:39.068706 5202 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:39.068900 5202 heartbeater.cc:511] Master 127.27.109.190:35881 requested a full tablet report, sending...
I20260928 02:44:39.069270 4883 ts_manager.cc:194] Registered new tserver with Master: 0aeb842bf059447f91b2f1ab320ea157 (127.27.109.130:44295)
I20260928 02:44:39.069527 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5074
I20260928 02:44:39.069620 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:39.069769 4883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.130:43665
I20260928 02:44:39.069813 28086 external_mini_cluster.cc:1428] Setting key 8c536674fd8b909ec8afa1864e8dcd3f
I20260928 02:44:39.070676 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--builtin_ntp_servers=127.27.109.148:41341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:44:39.150548 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.150743 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.150764 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.151906 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.151949 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.151973 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:39.153295 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.153533 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.153823 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.158716 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.158742 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.158876 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.158913 5205 server_base.cc:1034] running on GCE node
I20260928 02:44:39.159126 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.159401 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.160552 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790563479160529 us; error 38 us; skew 500 ppm
I20260928 02:44:39.161847 5205 webserver.cc:460] Webserver started at http://127.27.109.131:37707/ using document root <none> and password file <none>
I20260928 02:44:39.162086 5205 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.162133 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.162284 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.163403 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "6a737f5200c34df3b7ae03b0419bad05"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "2b28ae5cba82beecfda7e9af1b6ced82"
server_key_iv: "ac93050eaef4c2e09fbb81a8bc414f49"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.163756 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "6a737f5200c34df3b7ae03b0419bad05"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "2b28ae5cba82beecfda7e9af1b6ced82"
server_key_iv: "ac93050eaef4c2e09fbb81a8bc414f49"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.164954 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:44:39.165689 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.165876 5205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:39.165971 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "6a737f5200c34df3b7ae03b0419bad05"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "2b28ae5cba82beecfda7e9af1b6ced82"
server_key_iv: "ac93050eaef4c2e09fbb81a8bc414f49"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.166059 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.174592 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.174867 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.175012 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.175220 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:39.175509 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:39.175565 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.175611 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:39.175639 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.181315 5205 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:38277
I20260928 02:44:39.181360 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:38277 every 8 connection(s)
I20260928 02:44:39.181650 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:39.185946 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5205
I20260928 02:44:39.186061 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:39.186247 28086 external_mini_cluster.cc:1428] Setting key 0102847690a894c6d78dc3853146c7a8
I20260928 02:44:39.186520 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.182299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:35307 (local address 127.27.109.190:35881)
0928 02:44:39.182385 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.182388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.182637 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:44:39.183063 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.183064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.183074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.183125 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.183747 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.184216 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.185127 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.185301 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.185350 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:39.186184 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.186186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.186186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.186191 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.186207 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.186211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.186255 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.186394 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.186419 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:44:39.186731 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.182258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35881 (local address 127.27.109.131:35307)
0928 02:44:39.182439 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:39.182552 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:39.182987 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:39.183173 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:39.183176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:39.183183 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:39.183649 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:39.183653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.184293 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.184294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:39.185053 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:39.185054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:39.185370 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:39.185371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:39.185479 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:39.186028 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:39.186029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:39.186074 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:39.186104 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:39.186266 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:39.186270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:39.186273 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:39.186391 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:44:39.186504 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260928 02:44:39.187430 5333 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35881
I20260928 02:44:39.187526 5333 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:39.187666 5333 heartbeater.cc:511] Master 127.27.109.190:35881 requested a full tablet report, sending...
I20260928 02:44:39.187984 4882 ts_manager.cc:194] Registered new tserver with Master: 6a737f5200c34df3b7ae03b0419bad05 (127.27.109.131:38277)
I20260928 02:44:39.188318 4882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.131:35307
I20260928 02:44:39.198110 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:44:39.199970 4922 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:39.199469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.27.109.190:35881)
0928 02:44:39.199525 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.199528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.199639 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:44:39.199715 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.199716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.199726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.199802 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.199928 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35306: BlockingRecv error: recv got EOF from 127.0.0.1:35306 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:44:39.200268 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4943
I20260928 02:44:39.205485 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5074
I20260928 02:44:39.209211 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5205
I20260928 02:44:39.213238 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 4852
2026-09-28T02:44:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (560 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:39Z Disabled control of system clock
I20260928 02:44:39.239740 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36829
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:44:39.318099 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.318280 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.318301 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.319427 5344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:39.319468 5344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.319479 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.319492 5344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:39.319543 5344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:39.320820 5344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:36829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.321059 5344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.321332 5344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.325843 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.325840 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.325863 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.326188 5344 server_base.cc:1034] running on GCE node
I20260928 02:44:39.326406 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.326704 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.327880 5344 hybrid_clock.cc:648] HybridClock initialized: now 1790563479327861 us; error 32 us; skew 500 ppm
I20260928 02:44:39.329033 5344 webserver.cc:460] Webserver started at http://127.27.109.190:42223/ using document root <none> and password file <none>
I20260928 02:44:39.329267 5344 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.329334 5344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.329483 5344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.330530 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "eaee7035ec3b7c463a320a05838ba81f"
server_key_iv: "6656e9a73e6724d041c15f5df995de6b"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.330883 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "eaee7035ec3b7c463a320a05838ba81f"
server_key_iv: "6656e9a73e6724d041c15f5df995de6b"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.332134 5344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:44:39.332872 5358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.333045 5344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:39.333124 5344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "eaee7035ec3b7c463a320a05838ba81f"
server_key_iv: "6656e9a73e6724d041c15f5df995de6b"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.333194 5344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.337013 5344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.337206 5344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.337298 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.340770 5344 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:36829
I20260928 02:44:39.340798 5410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:36829 every 8 connection(s)
I20260928 02:44:39.341113 5344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:39.342051 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:39.343395 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5344
I20260928 02:44:39.343521 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:44:39.343743 28086 external_mini_cluster.cc:1428] Setting key c0c45a1fc611566c1018202fa9a18235
I20260928 02:44:39.344584 5411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3: Bootstrap starting.
I20260928 02:44:39.345230 5411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:39.345568 5411 log.cc:824] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:39.346141 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.344239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.27.109.190:36829)
0928 02:44:39.344418 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.344424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.344447 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:44:39.344852 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.344859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.344896 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.345008 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.345710 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.345715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.345715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.345724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.345762 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.345771 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.345869 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.345924 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.345980 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260928 02:44:39.346242 5411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3: No bootstrap required, opened a new log
I20260928 02:44:39.347710 5411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } }
I20260928 02:44:39.347838 5411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:39.347862 5411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2b7307a9d94b1eb1eb32b36e3768f3, State: Initialized, Role: FOLLOWER
I20260928 02:44:39.347978 5411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } }
I20260928 02:44:39.348074 5411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:39.348116 5411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:39.348161 5411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:39.348881 5411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } }
I20260928 02:44:39.348989 5411 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2b7307a9d94b1eb1eb32b36e3768f3; no voters:
I20260928 02:44:39.349170 5411 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:39.349228 5416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:39.349371 5416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d2b7307a9d94b1eb1eb32b36e3768f3, State: Running, Role: LEADER
I20260928 02:44:39.349421 5411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:39.349458 5416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } }
I20260928 02:44:39.349807 5417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } } }
I20260928 02:44:39.349823 5418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2b7307a9d94b1eb1eb32b36e3768f3. Latest consensus state: current_term: 1 leader_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2b7307a9d94b1eb1eb32b36e3768f3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36829 } } }
I20260928 02:44:39.350003 5418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:39.349963 5417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:39.350747 5425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:39.351248 5425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:44:39.352650 5425 catalog_manager.cc:1347] Generated new cluster ID: 0626bb644df94faebd1a8bdf407e6ebb
I20260928 02:44:39.352708 5425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:39.361235 5425 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:39.361387 5425 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:39.363397 5425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2b7307a9d94b1eb1eb32b36e3768f3: Generated new TSK 0
I20260928 02:44:39.363598 5425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:44:39.374001 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--builtin_ntp_servers=127.27.109.148:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:44:39.457193 5435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.457381 5435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.457402 5435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.458619 5435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.458657 5435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.458683 5435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:39.459995 5435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.460242 5435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.460538 5435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.466120 5440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.466128 5441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.466135 5443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.466634 5435 server_base.cc:1034] running on GCE node
I20260928 02:44:39.466849 5435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.467088 5435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.468272 5435 hybrid_clock.cc:648] HybridClock initialized: now 1790563479468258 us; error 30 us; skew 500 ppm
I20260928 02:44:39.469842 5435 webserver.cc:460] Webserver started at http://127.27.109.129:43667/ using document root <none> and password file <none>
I20260928 02:44:39.470108 5435 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.470181 5435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.470340 5435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.471441 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "36ed7898021e4308973bcf175e6b6497"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "de5ee1a2897453f6c4253c6a4120505f"
server_key_iv: "11cd3d50a70c3cb37b440da9e9181db7"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.471803 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "36ed7898021e4308973bcf175e6b6497"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "de5ee1a2897453f6c4253c6a4120505f"
server_key_iv: "11cd3d50a70c3cb37b440da9e9181db7"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.473121 5435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:44:39.473909 5449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.474081 5435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:39.474177 5435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "36ed7898021e4308973bcf175e6b6497"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "de5ee1a2897453f6c4253c6a4120505f"
server_key_iv: "11cd3d50a70c3cb37b440da9e9181db7"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.474259 5435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.477494 5435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.477743 5435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.477880 5435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.478084 5435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:39.478407 5435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:39.478451 5435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.478509 5435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:39.478540 5435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.483888 5435 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:38495
I20260928 02:44:39.483966 5562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:38495 every 8 connection(s)
I20260928 02:44:39.484239 5435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:39.487002 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.484928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:34217 (local address 127.27.109.190:36829)
0928 02:44:39.484972 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.484974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.485184 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:44:39.485650 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.485651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.485656 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.485707 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.486816 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.486817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.486817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.486821 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.486836 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.486840 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.486886 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.486938 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.486949 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:44:39.487186 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.484934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36829 (local address 127.27.109.129:34217)
0928 02:44:39.485090 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:39.485121 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:39.485552 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:39.485759 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:39.485762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:39.485769 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:39.486682 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:39.486682 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:39.486710 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:39.486738 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:39.486884 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:39.486887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:39.486889 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:44:39.486938 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:44:39.486982 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260928 02:44:39.487682 5563 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36829
I20260928 02:44:39.487754 5563 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:39.487901 5563 heartbeater.cc:511] Master 127.27.109.190:36829 requested a full tablet report, sending...
I20260928 02:44:39.488490 5375 ts_manager.cc:194] Registered new tserver with Master: 36ed7898021e4308973bcf175e6b6497 (127.27.109.129:38495)
I20260928 02:44:39.488641 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5435
I20260928 02:44:39.488734 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:39.488952 28086 external_mini_cluster.cc:1428] Setting key f474cb88a35e79dcee0f16406b0a7a75
I20260928 02:44:39.489899 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--builtin_ntp_servers=127.27.109.148:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:44:39.570180 5566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.570400 5566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.570434 5566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.571578 5566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.571645 5566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.571694 5566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:39.573005 5566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.573271 5566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.573550 5566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.578339 5574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.578405 5572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.578613 5566 server_base.cc:1034] running on GCE node
W20260928 02:44:39.578341 5571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.579661 5566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.579924 5566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.581063 5566 hybrid_clock.cc:648] HybridClock initialized: now 1790563479581054 us; error 28 us; skew 500 ppm
I20260928 02:44:39.582247 5566 webserver.cc:460] Webserver started at http://127.27.109.130:38183/ using document root <none> and password file <none>
I20260928 02:44:39.582535 5566 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.582582 5566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.582743 5566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.583761 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "7f7363d94a38474d823baed436545075"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "b31c1c812e0ef0b1a01378a7ea90819c"
server_key_iv: "2d22208e2a8b3008e764bd55c9800119"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.584107 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "7f7363d94a38474d823baed436545075"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "b31c1c812e0ef0b1a01378a7ea90819c"
server_key_iv: "2d22208e2a8b3008e764bd55c9800119"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.585345 5566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:39.586040 5580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.586205 5566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:39.586299 5566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "7f7363d94a38474d823baed436545075"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "b31c1c812e0ef0b1a01378a7ea90819c"
server_key_iv: "2d22208e2a8b3008e764bd55c9800119"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.586387 5566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.589604 5566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.589856 5566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.589991 5566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.590196 5566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:39.590523 5566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:39.590567 5566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.590597 5566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:39.590624 5566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.596221 5566 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:34677
I20260928 02:44:39.596263 5693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:34677 every 8 connection(s)
I20260928 02:44:39.596545 5566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:39.599392 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.597216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:35071 (local address 127.27.109.190:36829)
0928 02:44:39.597264 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.597266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.597500 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:44:39.597917 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.597918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.597923 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.597971 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.599215 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.599216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.599216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.599219 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.599229 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.599232 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.599275 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.599328 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.599340 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:44:39.599599 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.597216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36829 (local address 127.27.109.130:35071)
0928 02:44:39.597387 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:39.597418 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:39.597839 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:39.598028 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:39.598031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:39.598038 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:39.599088 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:39.599088 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:39.599115 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:39.599141 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:39.599274 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:39.599276 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:39.599279 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:39.599328 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:44:39.599372 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 02:44:39.600095 5694 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36829
I20260928 02:44:39.600203 5694 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:39.600360 5694 heartbeater.cc:511] Master 127.27.109.190:36829 requested a full tablet report, sending...
I20260928 02:44:39.600704 5375 ts_manager.cc:194] Registered new tserver with Master: 7f7363d94a38474d823baed436545075 (127.27.109.130:34677)
I20260928 02:44:39.604357 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5566
I20260928 02:44:39.604444 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:39.604638 28086 external_mini_cluster.cc:1428] Setting key 993636ab0424da9b8a39528dc0baabb6
I20260928 02:44:39.605551 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--builtin_ntp_servers=127.27.109.148:34983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:44:39.689844 5697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:39.690061 5697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:39.690083 5697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:39.691331 5697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:39.691370 5697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:39.691396 5697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:39.692725 5697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:39.692973 5697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:39.693267 5697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:39.698189 5705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:39.698148 5703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.698294 5697 server_base.cc:1034] running on GCE node
W20260928 02:44:39.698385 5702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:39.698609 5697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:39.698858 5697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:39.699994 5697 hybrid_clock.cc:648] HybridClock initialized: now 1790563479699979 us; error 30 us; skew 500 ppm
I20260928 02:44:39.701280 5697 webserver.cc:460] Webserver started at http://127.27.109.131:42719/ using document root <none> and password file <none>
I20260928 02:44:39.701514 5697 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:39.701561 5697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:39.701728 5697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:39.702854 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "242a8753788a4e1982ff3ef2c071feb3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "6aee9e27c0be3a7301962756289122e9"
server_key_iv: "ad1043cddf6b31ab38bb4b7dd4355a3e"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.703207 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "242a8753788a4e1982ff3ef2c071feb3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "6aee9e27c0be3a7301962756289122e9"
server_key_iv: "ad1043cddf6b31ab38bb4b7dd4355a3e"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.704460 5697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:44:39.705155 5711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.705317 5697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:39.705420 5697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "242a8753788a4e1982ff3ef2c071feb3"
format_stamp: "Formatted at 2026-09-28 02:44:39 on dist-test-slave-b5p5"
server_key: "6aee9e27c0be3a7301962756289122e9"
server_key_iv: "ad1043cddf6b31ab38bb4b7dd4355a3e"
server_key_version: "encryptionkey@0"
I20260928 02:44:39.705506 5697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:39.708812 5697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:39.709061 5697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:39.709210 5697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:39.709420 5697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:39.709719 5697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:39.709779 5697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.709818 5697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:39.709846 5697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:39.715533 5697 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:40263
I20260928 02:44:39.715571 5824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:40263 every 8 connection(s)
I20260928 02:44:39.715921 5697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:39.718945 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.716661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:45909 (local address 127.27.109.190:36829)
0928 02:44:39.716716 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.716718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.716950 (+ 232us) server_negotiation.cc:407] Connection header received
0928 02:44:39.717438 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.717439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.717445 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.717501 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.718666 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:39.718668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:39.718668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:44:39.718673 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:39.718687 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:44:39.718692 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:39.718749 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:39.718855 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:44:39.718870 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:44:39.719129 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:39.716657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36829 (local address 127.27.109.131:45909)
0928 02:44:39.716838 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:39.716870 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:39.717334 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:39.717573 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:39.717576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:39.717583 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:39.718455 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:44:39.718456 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:39.718529 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:44:39.718556 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:39.718753 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:39.718772 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:39.718775 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:39.718864 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:44:39.718927 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 02:44:39.719651 5825 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36829
I20260928 02:44:39.719723 5825 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:39.719874 5825 heartbeater.cc:511] Master 127.27.109.190:36829 requested a full tablet report, sending...
I20260928 02:44:39.720180 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5697
I20260928 02:44:39.720248 5375 ts_manager.cc:194] Registered new tserver with Master: 242a8753788a4e1982ff3ef2c071feb3 (127.27.109.131:40263)
I20260928 02:44:39.720292 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:39.720521 28086 external_mini_cluster.cc:1428] Setting key 40c4b40dea9410592bbc0d7c02bb08c3
I20260928 02:44:39.721218 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:44:39.723002 5414 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:44:39.722531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.27.109.190:36829)
0928 02:44:39.722572 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:39.722574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:39.722709 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:44:39.722738 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:39.722739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:39.722745 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:39.722826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:39.722926 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41402: BlockingRecv error: recv got EOF from 127.0.0.1:41402 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:44:39.723083 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5435
I20260928 02:44:39.727890 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5566
I20260928 02:44:39.732201 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5697
I20260928 02:44:39.736968 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5344
2026-09-28T02:44:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (523 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:39 dist-test-slave-b5p5 krb5kdc[5835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:39 dist-test-slave-b5p5 krb5kdc[5835](info): set up 2 sockets
Sep 28 02:44:39 dist-test-slave-b5p5 krb5kdc[5835](info): commencing operation
krb5kdc: starting...
W20260928 02:44:40.770694 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:44:40 dist-test-slave-b5p5 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:44:40.881218 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36545
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:40.958933 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:40.959149 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:40.959182 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:40.960286 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:44:40.960350 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:40.960371 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:40.960392 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:44:40.960421 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:44:40.961866 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:36545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:40.962127 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:40.962343 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:40.967074 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:40.967084 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:40.967200 5848 server_base.cc:1034] running on GCE node
W20260928 02:44:40.967083 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:40.967487 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:40.967806 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:40.968961 5848 hybrid_clock.cc:648] HybridClock initialized: now 1790563480968950 us; error 44 us; skew 500 ppm
Sep 28 02:44:40 dist-test-slave-b5p5 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:40.971297 5848 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:44:40.971742 5848 webserver.cc:460] Webserver started at http://127.27.109.190:38787/ using document root <none> and password file <none>
I20260928 02:44:40.971966 5848 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:40.972031 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:40.972167 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:40.973147 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "24b85ed940754447872164528bb13cb6"
format_stamp: "Formatted at 2026-09-28 02:44:40 on dist-test-slave-b5p5"
server_key: "7946d84de63a10c31c01133bc4be5fa6"
server_key_iv: "9565b002290182b0314fb07f1d76f228"
server_key_version: "encryptionkey@0"
I20260928 02:44:40.973520 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "24b85ed940754447872164528bb13cb6"
format_stamp: "Formatted at 2026-09-28 02:44:40 on dist-test-slave-b5p5"
server_key: "7946d84de63a10c31c01133bc4be5fa6"
server_key_iv: "9565b002290182b0314fb07f1d76f228"
server_key_version: "encryptionkey@0"
I20260928 02:44:40.974875 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:44:40.975631 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:40.975806 5848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:44:40.975904 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "24b85ed940754447872164528bb13cb6"
format_stamp: "Formatted at 2026-09-28 02:44:40 on dist-test-slave-b5p5"
server_key: "7946d84de63a10c31c01133bc4be5fa6"
server_key_iv: "9565b002290182b0314fb07f1d76f228"
server_key_version: "encryptionkey@0"
I20260928 02:44:40.975989 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:40.985045 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:40.986802 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:40.986951 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:40.990907 5848 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:36545
I20260928 02:44:40.990936 5915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:36545 every 8 connection(s)
I20260928 02:44:40.991266 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:44:40.992172 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:40.994203 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: Bootstrap starting.
I20260928 02:44:40.994789 5916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:40.995096 5916 log.cc:824] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:40.995127 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5848
I20260928 02:44:40.995220 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:44:40.995427 28086 external_mini_cluster.cc:1428] Setting key 536cf267cc103ae9362b3911ee94758c
I20260928 02:44:40.995900 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: No bootstrap required, opened a new log
I20260928 02:44:40.997474 5916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } }
I20260928 02:44:40.997620 5916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:40.997694 5916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b85ed940754447872164528bb13cb6, State: Initialized, Role: FOLLOWER
I20260928 02:44:40.997836 5916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } }
I20260928 02:44:40.997923 5916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:44:40.997967 5916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:44:40.998013 5916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:40.998909 5916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } }
I20260928 02:44:40.999029 5916 leader_election.cc:302] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b85ed940754447872164528bb13cb6; no voters:
I20260928 02:44:40.999212 5916 leader_election.cc:288] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:44:40.999236 5921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:40.999343 5921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [term 1 LEADER]: Becoming Leader. State: Replica: 24b85ed940754447872164528bb13cb6, State: Running, Role: LEADER
I20260928 02:44:40.999455 5916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:44:40.999486 5921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } }
I20260928 02:44:41.000802 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b85ed940754447872164528bb13cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } } }
I20260928 02:44:41.000896 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:44:41.001081 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b85ed940754447872164528bb13cb6. Latest consensus state: current_term: 1 leader_uuid: "24b85ed940754447872164528bb13cb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b85ed940754447872164528bb13cb6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 36545 } } }
I20260928 02:44:41.001647 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6 [sys.catalog]: This master's current role is: LEADER
W20260928 02:44:41.001380 5936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:44:41.001747 5936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:44:41.001817 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:44:41.001997 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:44:40 dist-test-slave-b5p5 krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:41.004134 5938 catalog_manager.cc:1347] Generated new cluster ID: 3b0c76fe730f47c2b425b33f3dd4b40b
I20260928 02:44:41.004189 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:44:41.007741 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:40.995883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.27.109.190:36545)
0928 02:44:40.996056 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:40.996064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:40.996097 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:44:40.997466 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:40.997473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:40.997521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:40.997703 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:40.998381 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:40.999518 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.000610 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.001258 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.001364 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.004012 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.004015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.004016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.004027 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.005896 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.006252 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.006255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.006255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.006373 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.006573 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.006573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.006574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.006838 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.006994 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.007436 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.007558 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260928 02:44:41.011796 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:44:41.012244 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:44:41.017021 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: Generated new TSK 0
I20260928 02:44:41.017179 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:44:41.047240 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:41.145133 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:41.145320 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:41.145344 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:41.146559 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:41.146600 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:41.146625 5942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:44:41.148028 5942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:41.148253 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:41.148487 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:41.153267 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.153275 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.153429 5942 server_base.cc:1034] running on GCE node
W20260928 02:44:41.153293 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.153692 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:41.153873 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:41.155032 5942 hybrid_clock.cc:648] HybridClock initialized: now 1790563481155018 us; error 26 us; skew 500 ppm
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:41.157428 5942 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:44:41.157874 5942 webserver.cc:460] Webserver started at http://127.27.109.129:35737/ using document root <none> and password file <none>
I20260928 02:44:41.158084 5942 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:41.158147 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:41.158288 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:41.159309 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "0d6ab77a631e4de2ab60c398cb000c77"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "889c2073488afbe57b1922ca09f4b533"
server_key_iv: "52576fef3d0169dc180e7143f9601d8c"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.159683 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6ab77a631e4de2ab60c398cb000c77"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "889c2073488afbe57b1922ca09f4b533"
server_key_iv: "52576fef3d0169dc180e7143f9601d8c"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.160928 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:44:41.161639 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.161829 5942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:41.161924 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "0d6ab77a631e4de2ab60c398cb000c77"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "889c2073488afbe57b1922ca09f4b533"
server_key_iv: "52576fef3d0169dc180e7143f9601d8c"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.162009 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:41.168447 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:41.170157 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:41.170308 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:41.170560 5942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:41.170852 5942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:41.170907 5942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.170946 5942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:41.170974 5942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.176775 5942 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:34719
I20260928 02:44:41.176810 6070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:34719 every 8 connection(s)
I20260928 02:44:41.177127 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:44:41.181959 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 5942
I20260928 02:44:41.182070 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:44:41.182312 28086 external_mini_cluster.cc:1428] Setting key a2b60a5962a0d1cf513308e023de9f19
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:41.186820 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.177760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:46779 (local address 127.27.109.190:36545)
0928 02:44:41.177804 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.177806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.178008 (+ 202us) server_negotiation.cc:407] Connection header received
0928 02:44:41.178589 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.178590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.178606 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.178667 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.179605 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.180114 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.181103 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.181319 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.181406 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.183722 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.183723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.183724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.183733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.185415 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.185786 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.185787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.185788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.185851 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.186068 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.186069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.186070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.186216 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.186292 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.186618 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.186728 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:44:41.186952 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.177713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36545 (local address 127.27.109.129:46779)
0928 02:44:41.177870 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.177931 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.178412 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.178734 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.178737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.179329 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:41.179503 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.179507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.180213 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.180214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.180996 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.180998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.181414 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.181416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.181544 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.182041 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:41.182053 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:41.183634 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:41.185504 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.185509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.185511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.185713 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.185918 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.185918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.185919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.186007 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.186284 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:41.186288 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:41.186388 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.186590 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.186743 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260928 02:44:41.187391 6071 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36545
I20260928 02:44:41.187494 6071 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:41.187695 6071 heartbeater.cc:511] Master 127.27.109.190:36545 requested a full tablet report, sending...
I20260928 02:44:41.188238 5880 ts_manager.cc:194] Registered new tserver with Master: 0d6ab77a631e4de2ab60c398cb000c77 (127.27.109.129:34719)
I20260928 02:44:41.189167 5880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:46779
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:44:41.206126 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:41.284624 6076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:41.284816 6076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:41.284837 6076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:41.285894 6076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:41.285930 6076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:41.285956 6076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:44:41.287364 6076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:41.287528 6076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:41.287690 6076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:41.292356 6082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.292371 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.292515 6076 server_base.cc:1034] running on GCE node
W20260928 02:44:41.292562 6081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.292795 6076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:41.293028 6076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:41.294191 6076 hybrid_clock.cc:648] HybridClock initialized: now 1790563481294177 us; error 27 us; skew 500 ppm
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:41.296584 6076 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:44:41.296936 6076 webserver.cc:460] Webserver started at http://127.27.109.130:46815/ using document root <none> and password file <none>
I20260928 02:44:41.297099 6076 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:41.297137 6076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:41.297221 6076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:41.298136 6076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "3bd3d1d81ed2479e87c94a1fc95571b0"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "3d135af442ebf5e1ee6ce313beae0f2a"
server_key_iv: "3ff036ff646012c7d61aef09cbfd96ef"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.298422 6076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "3bd3d1d81ed2479e87c94a1fc95571b0"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "3d135af442ebf5e1ee6ce313beae0f2a"
server_key_iv: "3ff036ff646012c7d61aef09cbfd96ef"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.299755 6076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:44:41.300498 6091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.300644 6076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:41.300738 6076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "3bd3d1d81ed2479e87c94a1fc95571b0"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "3d135af442ebf5e1ee6ce313beae0f2a"
server_key_iv: "3ff036ff646012c7d61aef09cbfd96ef"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.300791 6076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:41.331185 6076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:41.333405 6076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:41.333515 6076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:41.333693 6076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:41.333926 6076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:41.333956 6076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.333976 6076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:41.333989 6076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.339565 6076 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:37501
I20260928 02:44:41.339643 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:37501 every 8 connection(s)
I20260928 02:44:41.339931 6076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:44:41.340612 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 6076
I20260928 02:44:41.340736 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:44:41.340965 28086 external_mini_cluster.cc:1428] Setting key 173970de68c1dfcbc446c93994842500
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:44:41.351032 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.341534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:35301 (local address 127.27.109.190:36545)
0928 02:44:41.341608 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.341610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.341886 (+ 276us) server_negotiation.cc:407] Connection header received
0928 02:44:41.342632 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.342633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.342646 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.342718 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.343910 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.344536 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.345605 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.345923 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.345995 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.348273 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.348275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.348275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.348282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.349849 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.350213 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.350214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.350214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.350261 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.350435 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.350435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.350436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.350573 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.350652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.350891 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.350970 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:44:41.351156 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.341517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36545 (local address 127.27.109.130:35301)
0928 02:44:41.341740 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.341786 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.342455 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.342773 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.342777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.343585 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:41.343806 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.343811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.344626 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.344627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.345491 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.345493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.346011 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.346012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.346105 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.346639 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:41.346655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:41.348172 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:41.349936 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.349941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.349943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.350140 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.350307 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.350307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.350307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.350386 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.350658 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:41.350662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:41.350766 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.350852 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.350964 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260928 02:44:41.351698 6205 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36545
I20260928 02:44:41.351785 6205 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:41.351970 6205 heartbeater.cc:511] Master 127.27.109.190:36545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
I20260928 02:44:41.352380 5880 ts_manager.cc:194] Registered new tserver with Master: 3bd3d1d81ed2479e87c94a1fc95571b0 (127.27.109.130:37501)
I20260928 02:44:41.352759 5880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:35301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:44:41.365197 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:44:41.447140 6210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:44:41.447337 6210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:44:41.447359 6210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:44:41.448521 6210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:44:41.448565 6210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:44:41.448590 6210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:44:41.450016 6210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:36545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:44:41.450253 6210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:44:41.450495 6210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:44:41.455276 6216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.455232 6218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:44:41.455232 6215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:44:41.455499 6210 server_base.cc:1034] running on GCE node
I20260928 02:44:41.455695 6210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:44:41.455927 6210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:44:41.457064 6210 hybrid_clock.cc:648] HybridClock initialized: now 1790563481457042 us; error 37 us; skew 500 ppm
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:44:41.459424 6210 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:44:41.459851 6210 webserver.cc:460] Webserver started at http://127.27.109.131:39159/ using document root <none> and password file <none>
I20260928 02:44:41.460024 6210 fs_manager.cc:359] Metadata directory not provided
I20260928 02:44:41.460063 6210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:44:41.460150 6210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:44:41.461050 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "318c365d1604d70fce687cc63edd9b63"
server_key_iv: "176f0099b1d93251f148a670ae6a5428"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.461385 6210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "318c365d1604d70fce687cc63edd9b63"
server_key_iv: "176f0099b1d93251f148a670ae6a5428"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.462600 6210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:44:41.463263 6225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.463429 6210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:44:41.463527 6210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca"
format_stamp: "Formatted at 2026-09-28 02:44:41 on dist-test-slave-b5p5"
server_key: "318c365d1604d70fce687cc63edd9b63"
server_key_iv: "176f0099b1d93251f148a670ae6a5428"
server_key_version: "encryptionkey@0"
I20260928 02:44:41.463604 6210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:44:41.478792 6210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:44:41.480576 6210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:44:41.480731 6210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:44:41.480933 6210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:44:41.481223 6210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:44:41.481281 6210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.481319 6210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:44:41.481348 6210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:44:41.487093 6210 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:35571
I20260928 02:44:41.487138 6338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:35571 every 8 connection(s)
I20260928 02:44:41.487447 6210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:44:41.489451 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 6210
I20260928 02:44:41.489569 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:44:41.489774 28086 external_mini_cluster.cc:1428] Setting key 1ba61c773c2efd25e44256ec14f7b149
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:44:41.496236 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.488037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:39671 (local address 127.27.109.190:36545)
0928 02:44:41.488091 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.488093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.488323 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:44:41.488824 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.488825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.488838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.488886 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.489840 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.490587 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.491545 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.491757 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.491842 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.493729 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.493731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.493731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.493738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.494928 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.495332 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.495334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.495334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.495395 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.495627 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.495628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.495629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.495748 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.495818 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.496076 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.496150 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:44:41.496415 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.488047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:36545 (local address 127.27.109.131:39671)
0928 02:44:41.488210 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.488243 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.488721 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.488953 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.488963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.489491 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:44:41.489750 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.489756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.490659 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.490660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.491452 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.491454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.491851 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.491852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.491975 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.492364 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:44:41.492371 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:44:41.493634 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:44:41.495024 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.495030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.495032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.495243 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.495459 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:44:41.495460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:44:41.495461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:44:41.495559 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:44:41.495819 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:44:41.495825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:44:41.495948 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.496044 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.496198 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260928 02:44:41.496877 6339 heartbeater.cc:347] Connected to a master server at 127.27.109.190:36545
I20260928 02:44:41.496948 6339 heartbeater.cc:464] Registering TS with master...
I20260928 02:44:41.497124 6339 heartbeater.cc:511] Master 127.27.109.190:36545 requested a full tablet report, sending...
I20260928 02:44:41.497468 5880 ts_manager.cc:194] Registered new tserver with Master: 93cb2ef3b80f447bb1bd68b7f0a4a2ca (127.27.109.131:35571)
I20260928 02:44:41.497795 5880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:39671
I20260928 02:44:41.501487 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:44:41.508754 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.502871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.27.109.190:36545)
0928 02:44:41.502930 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.502932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.503010 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:44:41.503088 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.503088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.503098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.503146 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:44:41.503803 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.504260 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.505040 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.505250 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.505335 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.506494 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:44:41.506495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:44:41.506495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:44:41.506502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:44:41.507654 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.507986 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.507987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.507987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.508040 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:44:41.508246 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:44:41.508247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:44:41.508247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:44:41.508381 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:44:41.508490 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.508592 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.508663 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:44:41.510113 5880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:44:41.511709 5880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:44:41.518091 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.513959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.27.109.131:35571)
0928 02:44:41.514419 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.514423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.514432 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:44:41.514484 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.514486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.514502 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.514757 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.514802 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.515647 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517044 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.517672 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517760 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.517812 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.517977 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.518012 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":94,"threads_started":1}
I20260928 02:44:41.518092 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.513783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:35571 (local address 127.0.0.1:46218)
0928 02:44:41.514161 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.514227 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.514314 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.514615 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.514617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.514620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.514685 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.514686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.515760 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.515761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.516956 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.516958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.517759 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.517848 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.517861 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.517961 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.518012 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":59,"threads_started":1}
I20260928 02:44:41.518342 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.513977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:37501 (local address 127.0.0.1:44312)
0928 02:44:41.514812 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.514824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.514880 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.515043 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.515043 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.515045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.515080 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.515081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.516261 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.516262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.517242 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.517244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.518125 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.518125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.518190 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.518193 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.518236 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.518272 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":48,"threads_started":1}
I20260928 02:44:41.518636 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.514026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.27.109.129:34719)
0928 02:44:41.514565 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.514569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.514579 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:44:41.514612 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.514614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.514627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.514721 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.514836 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.515944 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517168 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.517893 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517967 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.518019 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.518513 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.518554 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":90,"threads_started":1}
I20260928 02:44:41.518633 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.514144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.27.109.130:37501)
0928 02:44:41.514347 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.514352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.514883 (+ 531us) server_negotiation.cc:407] Connection header received
0928 02:44:41.514936 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.514939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.514954 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.515033 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.515137 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.516162 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.517325 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.517994 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.518054 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.518097 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.518509 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.518553 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260928 02:44:41.518841 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.513802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34719 (local address 127.0.0.1:41688)
0928 02:44:41.514207 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.514237 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.514302 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.514713 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.514714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.514716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.514771 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.514773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.516046 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.516047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.517089 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.517090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.518055 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.518056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.518122 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.518126 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.518455 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.518782 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":54,"threads_started":1}
I20260928 02:44:41.518936 6273 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd94fe2bd6402ba50931824cf14549 (DEFAULT_TABLE table=test-table [id=02b06b5fcb9649678894470cf3a49f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.519299 6273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd94fe2bd6402ba50931824cf14549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.519263 6139 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd94fe2bd6402ba50931824cf14549 (DEFAULT_TABLE table=test-table [id=02b06b5fcb9649678894470cf3a49f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.519575 6139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd94fe2bd6402ba50931824cf14549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.519510 6005 tablet_service.cc:1467] Processing CreateTablet for tablet f0fd94fe2bd6402ba50931824cf14549 (DEFAULT_TABLE table=test-table [id=02b06b5fcb9649678894470cf3a49f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:44:41.519796 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fd94fe2bd6402ba50931824cf14549. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:44:41.521561 6353 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Bootstrap starting.
I20260928 02:44:41.522360 6353 tablet_bootstrap.cc:654] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.522815 6354 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Bootstrap starting.
I20260928 02:44:41.522996 6353 log.cc:824] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:41.523047 6355 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: Bootstrap starting.
I20260928 02:44:41.523631 6355 tablet_bootstrap.cc:654] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.523744 6353 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: No bootstrap required, opened a new log
I20260928 02:44:41.523818 6353 ts_tablet_manager.cc:1397] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:44:41.524286 6355 log.cc:824] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:41.524942 6355 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: No bootstrap required, opened a new log
I20260928 02:44:41.525019 6355 ts_tablet_manager.cc:1397] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:41.525151 6354 tablet_bootstrap.cc:654] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Neither blocks nor log segments found. Creating new log.
I20260928 02:44:41.525797 6354 log.cc:824] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:44:41.526085 6353 raft_consensus.cc:348] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.526245 6353 raft_consensus.cc:374] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.526310 6353 raft_consensus.cc:729] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cb2ef3b80f447bb1bd68b7f0a4a2ca, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.526443 6353 consensus_queue.cc:260] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.526623 6354 tablet_bootstrap.cc:492] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: No bootstrap required, opened a new log
I20260928 02:44:41.526697 6354 ts_tablet_manager.cc:1397] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:44:41.526739 6353 ts_tablet_manager.cc:1428] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:44:41.526866 6339 heartbeater.cc:503] Master 127.27.109.190:36545 was elected leader, sending a full tablet report...
I20260928 02:44:41.527076 6355 raft_consensus.cc:348] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.527233 6355 raft_consensus.cc:374] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.527312 6355 raft_consensus.cc:729] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6ab77a631e4de2ab60c398cb000c77, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.527424 6355 consensus_queue.cc:260] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.527686 6355 ts_tablet_manager.cc:1428] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:44:41.527765 6071 heartbeater.cc:503] Master 127.27.109.190:36545 was elected leader, sending a full tablet report...
I20260928 02:44:41.528723 6354 raft_consensus.cc:348] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.528837 6354 raft_consensus.cc:374] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:44:41.528857 6354 raft_consensus.cc:729] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd3d1d81ed2479e87c94a1fc95571b0, State: Initialized, Role: FOLLOWER
I20260928 02:44:41.528925 6354 consensus_queue.cc:260] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.529100 6354 ts_tablet_manager.cc:1428] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:44:41.529160 6205 heartbeater.cc:503] Master 127.27.109.190:36545 was elected leader, sending a full tablet report...
I20260928 02:44:41.566375 6361 raft_consensus.cc:482] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:44:41.566493 6361 raft_consensus.cc:504] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.566879 6361 leader_election.cc:288] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6ab77a631e4de2ab60c398cb000c77 (127.27.109.129:34719), 93cb2ef3b80f447bb1bd68b7f0a4a2ca (127.27.109.131:35571)
I20260928 02:44:41.570731 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.567129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42977 (local address 127.27.109.129:34719)
0928 02:44:41.567203 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.567206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.567352 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:44:41.567377 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.567377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.567387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.567457 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.567580 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.568259 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.569776 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.570349 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.570427 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.570434 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.570625 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.570663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:44:41.570731 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.567132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:38813 (local address 127.27.109.131:35571)
0928 02:44:41.567185 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.567188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.567197 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:44:41.567231 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.567232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.567240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.567290 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:44:41.567461 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.568228 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.569775 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.570352 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.570426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.570435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.570625 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.570663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:44:41.570763 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.567024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:35571 (local address 127.27.109.130:38813)
0928 02:44:41.567089 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.567102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.567147 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.567314 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.567315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.567317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.567392 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.567393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.568325 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.568326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.569687 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.569690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.570443 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.570444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.570545 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.570547 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.570606 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.570652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":70}
I20260928 02:44:41.570772 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.567036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34719 (local address 127.27.109.130:42977)
0928 02:44:41.567241 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.567255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:44:41.567319 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:44:41.567460 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:44:41.567461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:44:41.567462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:44:41.567520 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.567521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.568325 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.568326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.569698 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:44:41.569699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.570443 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.570444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:44:41.570542 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.570546 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:44:41.570606 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:44:41.570652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":101,"thread_start_us":136,"threads_started":1}
I20260928 02:44:41.571079 6293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd94fe2bd6402ba50931824cf14549" candidate_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" is_pre_election: true
I20260928 02:44:41.571270 6293 raft_consensus.cc:2454] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd3d1d81ed2479e87c94a1fc95571b0 in term 0.
I20260928 02:44:41.571507 6093 leader_election.cc:302] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd3d1d81ed2479e87c94a1fc95571b0, 93cb2ef3b80f447bb1bd68b7f0a4a2ca; no voters:
I20260928 02:44:41.571554 6025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd94fe2bd6402ba50931824cf14549" candidate_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6ab77a631e4de2ab60c398cb000c77" is_pre_election: true
I20260928 02:44:41.571700 6025 raft_consensus.cc:2454] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd3d1d81ed2479e87c94a1fc95571b0 in term 0.
I20260928 02:44:41.571729 6361 raft_consensus.cc:2781] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:44:41.571821 6361 raft_consensus.cc:482] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:44:41.571854 6361 raft_consensus.cc:3037] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.572544 6361 raft_consensus.cc:504] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.572642 6361 leader_election.cc:288] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [CANDIDATE]: Term 1 election: Requested vote from peers 0d6ab77a631e4de2ab60c398cb000c77 (127.27.109.129:34719), 93cb2ef3b80f447bb1bd68b7f0a4a2ca (127.27.109.131:35571)
I20260928 02:44:41.572796 6025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd94fe2bd6402ba50931824cf14549" candidate_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6ab77a631e4de2ab60c398cb000c77"
I20260928 02:44:41.572870 6025 raft_consensus.cc:3037] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.573565 6025 raft_consensus.cc:2454] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd3d1d81ed2479e87c94a1fc95571b0 in term 1.
I20260928 02:44:41.573676 6293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fd94fe2bd6402ba50931824cf14549" candidate_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca"
I20260928 02:44:41.573719 6092 leader_election.cc:302] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6ab77a631e4de2ab60c398cb000c77, 3bd3d1d81ed2479e87c94a1fc95571b0; no voters:
I20260928 02:44:41.573750 6293 raft_consensus.cc:3037] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:44:41.573854 6361 raft_consensus.cc:2781] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:44:41.574018 6361 raft_consensus.cc:686] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd3d1d81ed2479e87c94a1fc95571b0, State: Running, Role: LEADER
I20260928 02:44:41.574100 6361 consensus_queue.cc:237] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } }
I20260928 02:44:41.574649 6293 raft_consensus.cc:2454] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd3d1d81ed2479e87c94a1fc95571b0 in term 1.
I20260928 02:44:41.574868 5879 catalog_manager.cc:5495] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd3d1d81ed2479e87c94a1fc95571b0 (127.27.109.130). New cstate: current_term: 1 leader_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bd3d1d81ed2479e87c94a1fc95571b0" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 37501 } health_report { overall_health: HEALTHY } } }
I20260928 02:44:41.589821 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.587864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.27.109.130:37501)
0928 02:44:41.587913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:44:41.587915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:44:41.587929 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:44:41.587954 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:44:41.587955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:44:41.587969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:44:41.588040 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:44:41.588177 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.588598 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.589232 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:44:41.589415 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:44:41.589462 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:44:41.589597 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:44:41.589678 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:44:41.589732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:44:41.589761 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 02:44:41.589777 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 02:44:41.590556 6206 tablet.cc:2369] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:44:41.591468 6025 raft_consensus.cc:1261] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 1 FOLLOWER]: Refusing update from remote peer 3bd3d1d81ed2479e87c94a1fc95571b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:41.591475 6293 raft_consensus.cc:1261] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 1 FOLLOWER]: Refusing update from remote peer 3bd3d1d81ed2479e87c94a1fc95571b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:44:41.591781 6361 consensus_queue.cc:1035] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6ab77a631e4de2ab60c398cb000c77" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:41.591898 6363 consensus_queue.cc:1035] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93cb2ef3b80f447bb1bd68b7f0a4a2ca" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:44:41.594177 6366 mvcc.cc:205] Tried to move back new op lower bound from 7334148020596039680 to 7334148020528017408. Current Snapshot: MvccSnapshot[applied={T|T < 7334148020596039680}]
I20260928 02:44:41.595345 6369 mvcc.cc:205] Tried to move back new op lower bound from 7334148020596039680 to 7334148020528017408. Current Snapshot: MvccSnapshot[applied={T|T < 7334148020596039680}]
I20260928 02:44:41.596637 5879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:41.596722 5879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:44:41.597431 5879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24b85ed940754447872164528bb13cb6: Sending DeleteTablet for 3 replicas of tablet f0fd94fe2bd6402ba50931824cf14549
I20260928 02:44:41.597784 6273 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd94fe2bd6402ba50931824cf14549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:46218
I20260928 02:44:41.597824 6005 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd94fe2bd6402ba50931824cf14549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:41688
I20260928 02:44:41.598001 6375 tablet_replica.cc:330] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: stopping tablet replica
I20260928 02:44:41.598001 6376 tablet_replica.cc:330] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77: stopping tablet replica
I20260928 02:44:41.598028 6139 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fd94fe2bd6402ba50931824cf14549 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:44:41 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:44312
I20260928 02:44:41.598085 6376 raft_consensus.cc:2229] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:41.598085 6375 raft_consensus.cc:2229] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:44:41.598172 6376 raft_consensus.cc:2258] T f0fd94fe2bd6402ba50931824cf14549 P 0d6ab77a631e4de2ab60c398cb000c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.598184 6375 raft_consensus.cc:2258] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.598486 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5942
I20260928 02:44:41.599290 6377 tablet_replica.cc:330] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: stopping tablet replica
I20260928 02:44:41.599378 6377 raft_consensus.cc:2229] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:44:41.599462 6377 raft_consensus.cc:2258] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:44:41.600072 6377 ts_tablet_manager.cc:1905] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:41.600728 6375 ts_tablet_manager.cc:1905] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:44:41.601832 6377 ts_tablet_manager.cc:1918] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:41.601902 6377 log.cc:1196] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/f0fd94fe2bd6402ba50931824cf14549
I20260928 02:44:41.602173 6377 ts_tablet_manager.cc:1939] T f0fd94fe2bd6402ba50931824cf14549 P 3bd3d1d81ed2479e87c94a1fc95571b0: Deleting consensus metadata
I20260928 02:44:41.602392 6375 ts_tablet_manager.cc:1918] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:44:41.602485 6375 log.cc:1196] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/f0fd94fe2bd6402ba50931824cf14549
I20260928 02:44:41.602571 5865 catalog_manager.cc:4841] TS 3bd3d1d81ed2479e87c94a1fc95571b0 (127.27.109.130:37501): tablet f0fd94fe2bd6402ba50931824cf14549 (table test-table [id=02b06b5fcb9649678894470cf3a49f01]) successfully deleted
I20260928 02:44:41.602756 6375 ts_tablet_manager.cc:1939] T f0fd94fe2bd6402ba50931824cf14549 P 93cb2ef3b80f447bb1bd68b7f0a4a2ca: Deleting consensus metadata
I20260928 02:44:41.603075 5865 catalog_manager.cc:4841] TS 93cb2ef3b80f447bb1bd68b7f0a4a2ca (127.27.109.131:35571): tablet f0fd94fe2bd6402ba50931824cf14549 (table test-table [id=02b06b5fcb9649678894470cf3a49f01]) successfully deleted
W20260928 02:44:41.604981 5864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:34719 (error 108)
I20260928 02:44:41.605166 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 6076
I20260928 02:44:41.605788 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:44:41.605287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34719 (local address 127.0.0.1:41692)
0928 02:44:41.605714 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 02:44:41.605751 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:34719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409}
W20260928 02:44:41.610004 5864 catalog_manager.cc:4584] TS 0d6ab77a631e4de2ab60c398cb000c77 (127.27.109.129:34719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0fd94fe2bd6402ba50931824cf14549: Network error: Client connection negotiation failed: client connection to 127.27.109.129:34719: connect: Connection refused (error 111)
I20260928 02:44:41.610201 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 6210
I20260928 02:44:41.614449 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 5848
2026-09-28T02:44:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1878 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[6381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[6381](info): set up 2 sockets
Sep 28 02:44:41 dist-test-slave-b5p5 krb5kdc[6381](info): commencing operation
krb5kdc: starting...
W20260928 02:44:42.646729 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:44:42 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:44:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:44:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangeradmin_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangeradmin_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangerlookup_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangerlookup_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
I20260928 02:44:42.801182 28086 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpfCjkR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/postgres -l logfile start
2026-09-28 02:44:45.120 UTC [6411] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:44:45.120 UTC [6411] LOG: listening on IPv4 address "127.27.109.148", port 45769
2026-09-28 02:44:45.124 UTC [6411] LOG: listening on Unix socket "/tmp/.s.PGSQL.45769"
2026-09-28 02:44:45.157 UTC [6414] LOG: database system was shut down at 2026-09-28 02:44:43 UTC
2026-09-28 02:44:45.161 UTC [6411] LOG: database system is ready to accept connections
I20260928 02:44:46.110425 28086 mini_postgres.cc:96] Postgres bound to 45769
2026-09-28 02:44:46.116 UTC [6422] FATAL: database "slave" does not exist
127.27.109.148:45769 - accepting connections
I20260928 02:44:46.116786 28086 mini_ranger.cc:161] Starting Ranger...
I20260928 02:44:46.130661 28086 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:44:46.562041 28086 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:44:46.562130 28086 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin
2026-09-28 02:44:46,851 [I] DB FLAVOR :POSTGRES
2026-09-28 02:44:46,852 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:44:46,852 [I] Checking connection..
2026-09-28 02:44:46,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:44:47,105 [I] Checking connection passed.
2026-09-28 02:44:47,105 [I] --------- Verifying version history table ---------
2026-09-28 02:44:47,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:44:47,359 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:44:47,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:44:47,618 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:44:47,618 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:44:47,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:44:47.865 UTC [6486] WARNING: there is no transaction in progress
2026-09-28 02:44:47,871 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:44:47,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:44:48,090 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:44:48,091 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:44:48,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:44:48,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:44:48,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b5p5.c.gcp-upstream.internal','N') ;"
2026-09-28 02:44:48,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:44:49,058 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:44:49,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:44:49,304 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:44:49,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:44:49,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:44:49,792 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:44:49,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:44:50.711 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.727 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.742 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.757 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.775 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.791 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.842 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.848 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.861 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.871 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.880 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.893 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.902 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.908 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.914 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:50.920 UTC [6594] WARNING: there is no transaction in progress
2026-09-28 02:44:51,492 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:44:51,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:44:51,705 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:44:51,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:44:51,957 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:44:51,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:44:52,181 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:44:52,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:44:52,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:44:52,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b5p5.c.gcp-upstream.internal';"
2026-09-28 02:44:52,913 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:44:52,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:44:53,162 [I] DB_PATCHES have already been applied
I20260928 02:44:53.168051 28086 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpfCjkR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:44:53.168118 28086 mini_ranger.cc:193] Using host: 127.27.109.148
I20260928 02:44:53.171083 28086 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.148:48921
Sep 28, 2026 2:44:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:44:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:44:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:44:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:44:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:44:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:44:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:44:54.302062 28086 mini_ranger.cc:160] Time spent starting Ranger: real 8.185s user 0.000s sys 0.009s
Sep 28, 2026 2:45:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:45:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:45:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:45:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:45:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:45:14.685356 28086 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangerkms_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/rangerkms_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:45:14 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563514, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:45:14.775880 28086 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:45:14.785593 28086 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:45:15.582713 28086 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:45:15.582803 28086 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms
I20260928 02:45:15.582825 28086 mini_ranger_kms.cc:227] Using postgres at 127.27.109.148:45769
2026-09-28 02:45:15,752 [I] DB FLAVOR :POSTGRES
2026-09-28 02:45:15,752 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:45:15,752 [I] Checking connection
2026-09-28 02:45:15,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:45:15,996 [I] connection success
2026-09-28 02:45:15,996 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:45:15,996 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:45:15,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:45:16,227 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:45:16,228 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:45:16,228 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:45:16,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:45769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:45:16,482 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:45:16.933830 28086 mini_ranger_kms.cc:310] Created kms service
I20260928 02:45:17.114779 28086 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:45:17.155027 28086 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:45:17 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563514, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:17.577783 28086 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:45:17.577907 28086 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:45:17.577929 28086 mini_ranger_kms.cc:240] Using host: 127.27.109.148
I20260928 02:45:17.581782 28086 mini_ranger_kms.cc:276] Ranger KMS PID: 6774
I20260928 02:45:17.581902 28086 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.148:34569
Sep 28, 2026 2:45:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:45:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:45:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:45:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:45:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:45:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:45:19.840763 28086 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.065s user 0.000s sys 0.152s
I20260928 02:45:19.840874 28086 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:45:19.840974 28086 mini_ranger_kms.cc:401] 127.27.109.148:34569/kms/v1/keys
Sep 28 02:45:20 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:45:20 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
Sep 28 02:45:20 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563520, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
Sep 28 02:45:22 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:45:23.021175 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46137
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:46137
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:34569/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:23.123456 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:23.123669 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:23.123692 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:23.124851 6825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:45:23.124889 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:23.124905 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:23.124919 6825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:45:23.124933 6825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:45:23.126638 6825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.148:34569/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:46137
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:45:23.126832 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:23.127000 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:23.131994 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.131973 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:23.132241 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:23.132181 6825 server_base.cc:1034] running on GCE node
I20260928 02:45:23.132498 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:23.132773 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:23.133934 6825 hybrid_clock.cc:648] HybridClock initialized: now 1790563523133929 us; error 37 us; skew 500 ppm
Sep 28 02:45:23 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563523, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:23.139092 6825 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:45:23.139477 6825 webserver.cc:460] Webserver started at http://127.27.109.190:45665/ using document root <none> and password file <none>
I20260928 02:45:23.139648 6825 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:23.139683 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:23.139766 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:45:23 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563523, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:23.183379 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "3f45f520d008465eb63c0023348528d6"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-b5p5"
server_key: "31a494345590ba36481fed0c179a25bb"
server_key_iv: "ed2b5ac3257361aa5248f9f1ac96feb7"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:23.183823 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "3f45f520d008465eb63c0023348528d6"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-b5p5"
server_key: "31a494345590ba36481fed0c179a25bb"
server_key_iv: "ed2b5ac3257361aa5248f9f1ac96feb7"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:23.185456 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:45:23.200979 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:23.201218 6825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:45:23.201332 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "3f45f520d008465eb63c0023348528d6"
format_stamp: "Formatted at 2026-09-28 02:45:23 on dist-test-slave-b5p5"
server_key: "31a494345590ba36481fed0c179a25bb"
server_key_iv: "ed2b5ac3257361aa5248f9f1ac96feb7"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:23.201432 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:23.229238 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:23.231093 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:23.231213 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:23.235504 6825 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:46137
I20260928 02:45:23.235548 6892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:46137 every 8 connection(s)
I20260928 02:45:23.235875 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:45:23.236521 6893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:23.237226 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 6825
I20260928 02:45:23.237346 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:45:23.237555 28086 external_mini_cluster.cc:1428] Setting key 1b8ebe1e7fba901c6235c7263db00f91
I20260928 02:45:23.239359 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: Bootstrap starting.
I20260928 02:45:23.239909 6893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:23.240170 6893 log.cc:824] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:23.240815 6893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: No bootstrap required, opened a new log
I20260928 02:45:23.243049 6893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } }
I20260928 02:45:23.243196 6893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:23.243242 6893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f45f520d008465eb63c0023348528d6, State: Initialized, Role: FOLLOWER
I20260928 02:45:23.243367 6893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } }
I20260928 02:45:23.243455 6893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:45:23.243498 6893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:45:23.243542 6893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 02:45:23 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:45:23.244062 6893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } }
I20260928 02:45:23.244175 6893 leader_election.cc:302] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f45f520d008465eb63c0023348528d6; no voters:
I20260928 02:45:23.244321 6893 leader_election.cc:288] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:45:23.244349 6898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:23.244470 6898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [term 1 LEADER]: Becoming Leader. State: Replica: 3f45f520d008465eb63c0023348528d6, State: Running, Role: LEADER
I20260928 02:45:23.244594 6893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:45:23.244601 6898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } }
I20260928 02:45:23.245167 6899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f45f520d008465eb63c0023348528d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } } }
I20260928 02:45:23.245252 6899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:23.245729 6900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f45f520d008465eb63c0023348528d6. Latest consensus state: current_term: 1 leader_uuid: "3f45f520d008465eb63c0023348528d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f45f520d008465eb63c0023348528d6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 46137 } } }
I20260928 02:45:23.245814 6900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:45:23.245972 6907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:45:23.246138 6893 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:45:23.246488 6907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:45:23.250805 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:23.238074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.27.109.190:46137)
0928 02:45:23.238261 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:23.238271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:23.238305 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:45:23.238749 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:23.238757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:23.238804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:23.238957 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:23.239609 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.240722 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.241522 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:23.241946 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:23.242059 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:23.244547 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:23.244551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:23.244551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:23.244561 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:23.248209 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:23.248787 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:23.248794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:23.248795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:23.249069 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:23.249322 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:23.249322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:23.249323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:23.249748 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:23.249892 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:23.250419 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 02:45:23.250603 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260928 02:45:23.251216 6916 mvcc.cc:205] Tried to move back new op lower bound from 7334148191235076096 to 7334148191210672128. Current Snapshot: MvccSnapshot[applied={T|T < 7334148191235076096}]
I20260928 02:45:23.251745 6907 catalog_manager.cc:1347] Generated new cluster ID: dcb0a66634c54f2d8d14063c5f8fecc5
I20260928 02:45:23.251788 6907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:45:23.257587 6907 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:45:23.258330 6907 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:45:23.266028 6907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: Generated new TSK 0
I20260928 02:45:23.266198 6907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:45:23.964811 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:34569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:24.070904 6938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:24.071118 6938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:24.071153 6938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:24.072407 6938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:24.072477 6938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:24.072520 6938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:45:24.073953 6938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.148:34569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:45:24.074211 6938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:24.074432 6938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:24.080173 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:24.080206 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:24.080423 6938 server_base.cc:1034] running on GCE node
W20260928 02:45:24.080484 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:24.080715 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:24.080945 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:24.082098 6938 hybrid_clock.cc:648] HybridClock initialized: now 1790563524082086 us; error 31 us; skew 500 ppm
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:24.085471 6938 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:45:24.085907 6938 webserver.cc:460] Webserver started at http://127.27.109.129:37923/ using document root <none> and password file <none>
I20260928 02:45:24.086201 6938 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:24.086263 6938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:24.086427 6938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:24.100129 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "aaf7b1f9c64f4648be8f052384b23ab9"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "24f818e32f4bff8b1b1447aa27f465b3"
server_key_iv: "2104ca771dd100a59059f352d825802b"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.100696 6938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "aaf7b1f9c64f4648be8f052384b23ab9"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "24f818e32f4bff8b1b1447aa27f465b3"
server_key_iv: "2104ca771dd100a59059f352d825802b"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.103147 6938 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:45:24.119544 6955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.119765 6938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:45:24.119850 6938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "aaf7b1f9c64f4648be8f052384b23ab9"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "24f818e32f4bff8b1b1447aa27f465b3"
server_key_iv: "2104ca771dd100a59059f352d825802b"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.119926 6938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:24.130596 6938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:24.132745 6938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:24.132922 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:24.133153 6938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:24.133538 6938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:24.133643 6938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.133744 6938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:24.133821 6938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.142242 6938 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:40963
I20260928 02:45:24.142735 6938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:45:24.146185 7068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:40963 every 8 connection(s)
I20260928 02:45:24.149873 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 6938
I20260928 02:45:24.149981 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:45:24.150164 28086 external_mini_cluster.cc:1428] Setting key 0ed232c90561d5a1313e6d800dde4f99
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:45:24.158494 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.147301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46137 (local address 127.27.109.129:39343)
0928 02:45:24.147769 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.147815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.148439 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.148812 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.148816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.149075 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:24.149368 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.149375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.150396 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.150397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.151697 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.151700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.152275 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.152276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.152434 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.153036 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:24.153047 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:24.154693 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:24.156811 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.156818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.156820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.157112 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.157382 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.157383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.157384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.157528 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.157815 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:24.157821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:24.157971 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.158078 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.158225 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":90,"threads_started":1}
I20260928 02:45:24.158874 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.147315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:39343 (local address 127.27.109.190:46137)
0928 02:45:24.147706 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.147710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.148583 (+ 873us) server_negotiation.cc:407] Connection header received
0928 02:45:24.148613 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.148614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.148628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.148689 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:24.149482 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.150283 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.151792 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.152149 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.152210 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.154804 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:24.154806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:24.154806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:24.154814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:24.156720 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.157207 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.157208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.157209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.157273 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.157606 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.157607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.157607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.157735 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:24.157791 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.158693 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.158780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":107,"threads_started":1}
I20260928 02:45:24.159422 7069 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46137
I20260928 02:45:24.159539 7069 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:24.159749 7069 heartbeater.cc:511] Master 127.27.109.190:46137 requested a full tablet report, sending...
I20260928 02:45:24.160333 6857 ts_manager.cc:194] Registered new tserver with Master: aaf7b1f9c64f4648be8f052384b23ab9 (127.27.109.129:40963)
I20260928 02:45:24.161232 6857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:39343
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:45:24.199393 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:34569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:24.296202 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:24.296407 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:24.296443 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:24.297725 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:24.297793 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:24.297837 7075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:45:24.299424 7075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.148:34569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:45:24.299672 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:24.299914 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:45:24.305699 7083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:24.305840 7082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:24.306048 7075 server_base.cc:1034] running on GCE node
W20260928 02:45:24.306852 7085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:24.307147 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:24.307413 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:24.308573 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790563524308546 us; error 41 us; skew 500 ppm
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:24.312139 7075 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:45:24.312673 7075 webserver.cc:460] Webserver started at http://127.27.109.130:44273/ using document root <none> and password file <none>
I20260928 02:45:24.312966 7075 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:24.313086 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:24.313257 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:24.329196 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "d344d16c85004a32bffcc9630aa47cce"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "c827fb37e17c56b0c9fcd445fafb6552"
server_key_iv: "ef996f83245eeb1e163702f1c94b2089"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.329617 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "d344d16c85004a32bffcc9630aa47cce"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "c827fb37e17c56b0c9fcd445fafb6552"
server_key_iv: "ef996f83245eeb1e163702f1c94b2089"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.338387 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.002s
I20260928 02:45:24.353878 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.355679 7075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260928 02:45:24.355782 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "d344d16c85004a32bffcc9630aa47cce"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "c827fb37e17c56b0c9fcd445fafb6552"
server_key_iv: "ef996f83245eeb1e163702f1c94b2089"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.355870 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:24.367677 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:24.370019 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:24.370203 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:24.370430 7075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:24.370857 7075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:24.370931 7075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.370997 7075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:24.371044 7075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.377831 7075 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:42601
I20260928 02:45:24.378194 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:42601 every 8 connection(s)
I20260928 02:45:24.378309 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:45:24.380079 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 7075
I20260928 02:45:24.380194 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:45:24.380430 28086 external_mini_cluster.cc:1428] Setting key e20dd11dcb567c9ae3d6fe6fd0d14f78
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:45:24.394229 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.381787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:45085 (local address 127.27.109.190:46137)
0928 02:45:24.381830 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.381832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.382694 (+ 862us) server_negotiation.cc:407] Connection header received
0928 02:45:24.383357 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.383358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.383373 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.383404 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:24.384260 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.385049 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.386585 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.386858 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.386905 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.389494 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:24.389495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:24.389496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:24.389503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:24.391159 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.391627 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.391628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.391628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.391695 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.393488 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.393489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.393490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.393622 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:24.393667 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.394046 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.394134 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:45:24.394996 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.381536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46137 (local address 127.27.109.130:45085)
0928 02:45:24.382565 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.382612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.383232 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.383599 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.383603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.383863 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:24.384145 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.384152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.385162 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.385163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.386445 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.386448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.386996 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.386997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.387127 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.387746 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:24.387756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:24.389367 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:24.391251 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.391257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.391259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.391537 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.393255 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.393256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.393256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.393390 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.393716 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:24.393722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:24.393883 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.394606 (+ 723us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.394769 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":79,"threads_started":1}
I20260928 02:45:24.395687 7207 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46137
I20260928 02:45:24.395783 7207 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:24.395977 7207 heartbeater.cc:511] Master 127.27.109.190:46137 requested a full tablet report, sending...
I20260928 02:45:24.396390 6857 ts_manager.cc:194] Registered new tserver with Master: d344d16c85004a32bffcc9630aa47cce (127.27.109.130:42601)
I20260928 02:45:24.396915 6857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:45085
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:45:24.419564 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:34569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:45:24.538270 7212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:45:24.538633 7212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:45:24.538702 7212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:45:24.540398 7212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:45:24.540607 7212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:45:24.540680 7212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:45:24.542819 7212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.148:34569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:45:24.543224 7212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:45:24.543550 7212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:45:24.549958 7218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:24.550309 7217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:45:24.550526 7220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:45:24.551508 7212 server_base.cc:1034] running on GCE node
I20260928 02:45:24.551842 7212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:45:24.552111 7212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:45:24.553257 7212 hybrid_clock.cc:648] HybridClock initialized: now 1790563524553227 us; error 42 us; skew 500 ppm
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:45:24.557010 7212 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:45:24.557611 7212 webserver.cc:460] Webserver started at http://127.27.109.131:41119/ using document root <none> and password file <none>
I20260928 02:45:24.557895 7212 fs_manager.cc:359] Metadata directory not provided
I20260928 02:45:24.557984 7212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:45:24.558133 7212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:24.575098 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "de987d4df3b9436f9238526faa87a342"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "ca4729e63a9076299fffc6ecbfa68da7"
server_key_iv: "ae094b002ee321e1e551e2377f0a87ad"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.575542 7212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "de987d4df3b9436f9238526faa87a342"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "ca4729e63a9076299fffc6ecbfa68da7"
server_key_iv: "ae094b002ee321e1e551e2377f0a87ad"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.579032 7212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:45:24.592869 7227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.593128 7212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:45:24.593240 7212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "de987d4df3b9436f9238526faa87a342"
format_stamp: "Formatted at 2026-09-28 02:45:24 on dist-test-slave-b5p5"
server_key: "ca4729e63a9076299fffc6ecbfa68da7"
server_key_iv: "ae094b002ee321e1e551e2377f0a87ad"
server_key_version: "kuduclusterkey@0"
I20260928 02:45:24.593341 7212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:45:24.624931 7212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:45:24.627434 7212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:45:24.627620 7212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:45:24.627882 7212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:45:24.628260 7212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:45:24.628340 7212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.628420 7212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:45:24.628482 7212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:45:24.635046 7212 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:36777
I20260928 02:45:24.635519 7212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:45:24.641570 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 7212
I20260928 02:45:24.641681 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:45:24.641862 28086 external_mini_cluster.cc:1428] Setting key e06d03cc10ba5c03b5d5ecc6958ca78d
I20260928 02:45:24.645512 7340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:36777 every 8 connection(s)
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:45:24.656599 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.646216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:42263 (local address 127.27.109.190:46137)
0928 02:45:24.646268 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.646270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.646591 (+ 321us) server_negotiation.cc:407] Connection header received
0928 02:45:24.647227 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.647228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.647240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.647270 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:24.648080 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.648791 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.650148 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.650418 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.650458 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.653488 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:24.653489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:24.653489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:24.653496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:24.655156 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.655621 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.655622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.655622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.655675 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.655922 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.655923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.655923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.656043 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:24.656082 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.656428 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.656507 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:45:24.657428 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.646224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:46137 (local address 127.27.109.131:42263)
0928 02:45:24.646427 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.646508 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.647108 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.647462 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.647466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.647703 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:45:24.647977 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.647983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.648903 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.648903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.650050 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.650053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.650563 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.650563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.650693 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.651262 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:45:24.651272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:45:24.653400 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:45:24.655250 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.655255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.655258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.655537 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.655740 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:45:24.655740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:45:24.655741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:45:24.655854 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:45:24.656124 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:45:24.656128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:45:24.656275 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.657003 (+ 728us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.657153 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260928 02:45:24.658021 7341 heartbeater.cc:347] Connected to a master server at 127.27.109.190:46137
I20260928 02:45:24.658105 7341 heartbeater.cc:464] Registering TS with master...
I20260928 02:45:24.658272 7341 heartbeater.cc:511] Master 127.27.109.190:46137 requested a full tablet report, sending...
I20260928 02:45:24.658674 6857 ts_manager.cc:194] Registered new tserver with Master: de987d4df3b9436f9238526faa87a342 (127.27.109.131:36777)
I20260928 02:45:24.659137 6857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:42263
I20260928 02:45:24.663983 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:45:24.678776 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.670843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.27.109.190:46137)
0928 02:45:24.670886 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.670888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.671173 (+ 285us) server_negotiation.cc:407] Connection header received
0928 02:45:24.671270 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.671271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.671282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.671310 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:45:24.672085 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.672861 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.673991 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.674284 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.674326 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.675881 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:45:24.675882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:45:24.675883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:45:24.675889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:45:24.677572 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.677947 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.677948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.677949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.677999 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:45:24.678177 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:45:24.678178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:45:24.678178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:45:24.678289 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:45:24.678369 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.678607 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.678681 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:45:24.680826 6857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:45:24.683094 6857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:45:24.698334 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.686710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.27.109.131:36777)
0928 02:45:24.687705 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.687708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.691319 (+ 3611us) server_negotiation.cc:407] Connection header received
0928 02:45:24.691368 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.691370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.691386 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.691448 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:24.691629 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.692583 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.696994 (+ 4411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.697808 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.697897 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.697955 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.698213 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.698254 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":76,"threads_started":1}
I20260928 02:45:24.698833 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.686364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.27.109.130:42601)
0928 02:45:24.687401 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.687406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.687415 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:45:24.687451 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.687453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.687467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.687530 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:24.687887 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.694271 (+ 6384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.695788 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.696597 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.696698 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.696759 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.698691 (+ 1932us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.698743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":88,"threads_started":1}
I20260928 02:45:24.699219 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.686574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.27.109.129:40963)
0928 02:45:24.687633 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.687637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.689058 (+ 1421us) server_negotiation.cc:407] Connection header received
0928 02:45:24.689093 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.689096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.689111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.689175 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:24.689366 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.690328 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.696296 (+ 5968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.698220 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.698295 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.698362 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.699079 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.699131 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":50,"threads_started":1}
I20260928 02:45:24.698021 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.685649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:42601 (local address 127.0.0.1:42372)
0928 02:45:24.687169 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.687208 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.687265 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.687744 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.687747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.687749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:24.687806 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.687808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.694369 (+ 6561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.694370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.695409 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.695411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.696931 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.696933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.697011 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.697341 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.697429 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.697545 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1492,"thread_start_us":65,"threads_started":1}
I20260928 02:45:24.699692 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.686000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:36777 (local address 127.0.0.1:60834)
0928 02:45:24.691215 (+ 5215us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.691224 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.691266 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.691515 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.691516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.691518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:24.691563 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.691564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.692681 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.692682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.696872 (+ 4190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.696876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.697992 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.697993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.698071 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.698075 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.698131 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.698169 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5200,"thread_start_us":50,"threads_started":1}
I20260928 02:45:24.701241 7275 tablet_service.cc:1467] Processing CreateTablet for tablet 30f63e89f3bf43529be8a186319feb81 (DEFAULT_TABLE table=test-table [id=fe8d9e2ab6bf4d309cfbe8d41f50fe55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:24.701601 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f63e89f3bf43529be8a186319feb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:24.699782 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.685845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40963 (local address 127.0.0.1:60188)
0928 02:45:24.688953 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.688962 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.689004 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.689233 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.689234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.689235 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:24.689289 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.689290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.690435 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.690436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.696206 (+ 5770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.696209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.698401 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.698402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.698504 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.698509 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.698566 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.698645 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3094,"thread_start_us":63,"threads_started":1}
I20260928 02:45:24.706307 7140 tablet_service.cc:1467] Processing CreateTablet for tablet 30f63e89f3bf43529be8a186319feb81 (DEFAULT_TABLE table=test-table [id=fe8d9e2ab6bf4d309cfbe8d41f50fe55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:24.706676 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f63e89f3bf43529be8a186319feb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:24.710450 7360 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Bootstrap starting.
I20260928 02:45:24.710697 7003 tablet_service.cc:1467] Processing CreateTablet for tablet 30f63e89f3bf43529be8a186319feb81 (DEFAULT_TABLE table=test-table [id=fe8d9e2ab6bf4d309cfbe8d41f50fe55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:45:24.711016 7359 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Bootstrap starting.
I20260928 02:45:24.711038 7003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f63e89f3bf43529be8a186319feb81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:45:24.711989 7359 tablet_bootstrap.cc:654] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:24.712682 7360 tablet_bootstrap.cc:654] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:24.713029 7360 log.cc:824] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:24.713414 7359 log.cc:824] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:24.714498 7360 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: No bootstrap required, opened a new log
I20260928 02:45:24.714620 7360 ts_tablet_manager.cc:1397] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 02:45:24.716908 7360 raft_consensus.cc:348] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.717164 7360 raft_consensus.cc:374] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:24.717247 7360 raft_consensus.cc:729] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d344d16c85004a32bffcc9630aa47cce, State: Initialized, Role: FOLLOWER
I20260928 02:45:24.717442 7360 consensus_queue.cc:260] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.717762 7360 ts_tablet_manager.cc:1428] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:45:24.717907 7207 heartbeater.cc:503] Master 127.27.109.190:46137 was elected leader, sending a full tablet report...
I20260928 02:45:24.719326 7359 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: No bootstrap required, opened a new log
I20260928 02:45:24.719492 7359 ts_tablet_manager.cc:1397] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 02:45:24.721308 7359 raft_consensus.cc:348] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.721521 7359 raft_consensus.cc:374] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:24.721591 7359 raft_consensus.cc:729] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de987d4df3b9436f9238526faa87a342, State: Initialized, Role: FOLLOWER
I20260928 02:45:24.721750 7359 consensus_queue.cc:260] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.722016 7359 ts_tablet_manager.cc:1428] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:45:24.722148 7341 heartbeater.cc:503] Master 127.27.109.190:46137 was elected leader, sending a full tablet report...
I20260928 02:45:24.723585 7363 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: Bootstrap starting.
I20260928 02:45:24.724323 7363 tablet_bootstrap.cc:654] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: Neither blocks nor log segments found. Creating new log.
I20260928 02:45:24.724722 7363 log.cc:824] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:45:24.728791 7363 tablet_bootstrap.cc:492] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: No bootstrap required, opened a new log
I20260928 02:45:24.728899 7363 ts_tablet_manager.cc:1397] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 02:45:24.731240 7363 raft_consensus.cc:348] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.731436 7363 raft_consensus.cc:374] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:45:24.731500 7363 raft_consensus.cc:729] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf7b1f9c64f4648be8f052384b23ab9, State: Initialized, Role: FOLLOWER
I20260928 02:45:24.731644 7363 consensus_queue.cc:260] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.731896 7363 ts_tablet_manager.cc:1428] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:45:24.732026 7069 heartbeater.cc:503] Master 127.27.109.190:46137 was elected leader, sending a full tablet report...
Sep 28 02:45:24 dist-test-slave-b5p5 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563524, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:45:24.817937 7365 raft_consensus.cc:482] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:45:24.818058 7365 raft_consensus.cc:504] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.818435 7365 leader_election.cc:288] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d344d16c85004a32bffcc9630aa47cce (127.27.109.130:42601), aaf7b1f9c64f4648be8f052384b23ab9 (127.27.109.129:40963)
I20260928 02:45:24.822870 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.818699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:42601 (local address 127.27.109.131:41329)
0928 02:45:24.819121 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.819133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.819181 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.819579 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.819580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.819583 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:24.819650 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.819652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.820590 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.820591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.821847 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.821849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.822629 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.822630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.822710 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.822714 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.822759 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.822794 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400}
I20260928 02:45:24.823546 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.819316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41329 (local address 127.27.109.130:42601)
0928 02:45:24.819402 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.819404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.819412 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:45:24.819439 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.819440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.819449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.819495 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:24.819720 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.820476 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.821957 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.822527 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.822591 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.822599 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.823422 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.823483 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:45:24.823824 7160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f63e89f3bf43529be8a186319feb81" candidate_uuid: "de987d4df3b9436f9238526faa87a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d344d16c85004a32bffcc9630aa47cce" is_pre_election: true
I20260928 02:45:24.823969 7160 raft_consensus.cc:2454] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de987d4df3b9436f9238526faa87a342 in term 0.
I20260928 02:45:24.824352 7230 leader_election.cc:302] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d344d16c85004a32bffcc9630aa47cce, de987d4df3b9436f9238526faa87a342; no voters:
I20260928 02:45:24.824529 7365 raft_consensus.cc:2781] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:45:24.824594 7365 raft_consensus.cc:482] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:45:24.824616 7365 raft_consensus.cc:3037] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:24.825481 7365 raft_consensus.cc:504] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.825603 7365 leader_election.cc:288] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [CANDIDATE]: Term 1 election: Requested vote from peers d344d16c85004a32bffcc9630aa47cce (127.27.109.130:42601), aaf7b1f9c64f4648be8f052384b23ab9 (127.27.109.129:40963)
I20260928 02:45:24.825959 7160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f63e89f3bf43529be8a186319feb81" candidate_uuid: "de987d4df3b9436f9238526faa87a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d344d16c85004a32bffcc9630aa47cce"
I20260928 02:45:24.826030 7160 raft_consensus.cc:3037] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:24.826838 7160 raft_consensus.cc:2454] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de987d4df3b9436f9238526faa87a342 in term 1.
I20260928 02:45:24.827319 7230 leader_election.cc:302] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d344d16c85004a32bffcc9630aa47cce, de987d4df3b9436f9238526faa87a342; no voters:
I20260928 02:45:24.827437 7365 raft_consensus.cc:2781] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:45:24.827587 7365 raft_consensus.cc:686] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 1 LEADER]: Becoming Leader. State: Replica: de987d4df3b9436f9238526faa87a342, State: Running, Role: LEADER
I20260928 02:45:24.827667 7365 consensus_queue.cc:237] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } }
I20260928 02:45:24.828621 6857 catalog_manager.cc:5495] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 reported cstate change: term changed from 0 to 1, leader changed from <none> to de987d4df3b9436f9238526faa87a342 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "de987d4df3b9436f9238526faa87a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de987d4df3b9436f9238526faa87a342" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 36777 } health_report { overall_health: HEALTHY } } }
I20260928 02:45:24.837782 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.818803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40963 (local address 127.27.109.131:48679)
0928 02:45:24.822947 (+ 4144us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.822957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:45:24.822993 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:45:24.834710 (+ 11717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:45:24.834711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:45:24.834713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:45:24.834755 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.834756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.835689 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.835690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.836768 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:45:24.836770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.837569 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.837570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:45:24.837630 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.837634 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:45:24.837677 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:45:24.837710 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4135,"thread_start_us":116,"threads_started":1}
I20260928 02:45:24.838102 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.819361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:48679 (local address 127.27.109.129:40963)
0928 02:45:24.819547 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.819549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.823044 (+ 3495us) server_negotiation.cc:407] Connection header received
0928 02:45:24.823072 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.823074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.823082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.823124 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:45:24.834834 (+ 11710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.835593 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.836864 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.837464 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.837531 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.837539 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.837994 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.838038 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260928 02:45:24.838641 7023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f63e89f3bf43529be8a186319feb81" candidate_uuid: "de987d4df3b9436f9238526faa87a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" is_pre_election: true
I20260928 02:45:24.838649 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f63e89f3bf43529be8a186319feb81" candidate_uuid: "de987d4df3b9436f9238526faa87a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf7b1f9c64f4648be8f052384b23ab9"
I20260928 02:45:24.838797 7023 raft_consensus.cc:2454] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de987d4df3b9436f9238526faa87a342 in term 0.
I20260928 02:45:24.848984 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.845656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.27.109.131:36777)
0928 02:45:24.845731 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:45:24.845733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:45:24.845740 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:45:24.845770 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:45:24.845771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:45:24.845789 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:45:24.845850 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:45:24.846048 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.846847 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.847956 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:45:24.848254 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:45:24.848313 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:45:24.848520 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:45:24.848646 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:45:24.848713 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:45:24.848873 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:45:24.848893 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:45:24.852500 7023 raft_consensus.cc:3037] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:45:24.853406 7023 raft_consensus.cc:1261] T 30f63e89f3bf43529be8a186319feb81 P aaf7b1f9c64f4648be8f052384b23ab9 [term 1 FOLLOWER]: Refusing update from remote peer de987d4df3b9436f9238526faa87a342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:24.853679 7160 raft_consensus.cc:1261] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 1 FOLLOWER]: Refusing update from remote peer de987d4df3b9436f9238526faa87a342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:45:24.854157 7365 consensus_queue.cc:1035] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf7b1f9c64f4648be8f052384b23ab9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:24.854298 7365 consensus_queue.cc:1035] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d344d16c85004a32bffcc9630aa47cce" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:45:24.865368 7372 mvcc.cc:205] Tried to move back new op lower bound from 7334148197792854016 to 7334148197694550016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148197792854016}]
I20260928 02:45:24.867216 7375 mvcc.cc:205] Tried to move back new op lower bound from 7334148197792854016 to 7334148197694550016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148197792854016}]
I20260928 02:45:24.868958 7373 mvcc.cc:205] Tried to move back new op lower bound from 7334148197792854016 to 7334148197694550016. Current Snapshot: MvccSnapshot[applied={T|T < 7334148197792854016}]
I20260928 02:45:24.871869 6857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:24.871969 6857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:45:24.872679 6857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f45f520d008465eb63c0023348528d6: Sending DeleteTablet for 3 replicas of tablet 30f63e89f3bf43529be8a186319feb81
I20260928 02:45:24.873153 7003 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f63e89f3bf43529be8a186319feb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:60188
I20260928 02:45:24.873185 7140 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f63e89f3bf43529be8a186319feb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:42372
I20260928 02:45:24.873399 7275 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f63e89f3bf43529be8a186319feb81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:45:24 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:60834
I20260928 02:45:24.874024 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 6938
I20260928 02:45:24.875667 7383 tablet_replica.cc:330] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: stopping tablet replica
I20260928 02:45:24.875833 7384 tablet_replica.cc:330] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: stopping tablet replica
I20260928 02:45:24.875980 7384 raft_consensus.cc:2229] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:45:24.876104 7383 raft_consensus.cc:2229] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:45:24.876261 7384 raft_consensus.cc:2258] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:24.876397 7383 raft_consensus.cc:2258] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:45:24.877019 7384 ts_tablet_manager.cc:1905] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:24.877372 7383 ts_tablet_manager.cc:1905] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:45:24.880834 7384 ts_tablet_manager.cc:1918] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:24.880977 7384 log.cc:1196] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/30f63e89f3bf43529be8a186319feb81
I20260928 02:45:24.881314 7384 ts_tablet_manager.cc:1939] T 30f63e89f3bf43529be8a186319feb81 P de987d4df3b9436f9238526faa87a342: Deleting consensus metadata
I20260928 02:45:24.882858 6844 catalog_manager.cc:4841] TS de987d4df3b9436f9238526faa87a342 (127.27.109.131:36777): tablet 30f63e89f3bf43529be8a186319feb81 (table test-table [id=fe8d9e2ab6bf4d309cfbe8d41f50fe55]) successfully deleted
I20260928 02:45:24.883090 7383 ts_tablet_manager.cc:1918] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:45:24.883200 7383 log.cc:1196] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/30f63e89f3bf43529be8a186319feb81
I20260928 02:45:24.883530 7383 ts_tablet_manager.cc:1939] T 30f63e89f3bf43529be8a186319feb81 P d344d16c85004a32bffcc9630aa47cce: Deleting consensus metadata
I20260928 02:45:24.883985 6843 catalog_manager.cc:4841] TS d344d16c85004a32bffcc9630aa47cce (127.27.109.130:42601): tablet 30f63e89f3bf43529be8a186319feb81 (table test-table [id=fe8d9e2ab6bf4d309cfbe8d41f50fe55]) successfully deleted
W20260928 02:45:24.897131 6842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:40963 (error 108)
I20260928 02:45:24.897976 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 7075
I20260928 02:45:24.902855 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 7212
I20260928 02:45:24.903573 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:45:24.903037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40963 (local address 127.0.0.1:60198)
0928 02:45:24.903482 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 02:45:24.903535 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:40963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
W20260928 02:45:24.904613 6842 catalog_manager.cc:4584] TS aaf7b1f9c64f4648be8f052384b23ab9 (127.27.109.129:40963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30f63e89f3bf43529be8a186319feb81: Network error: Client connection negotiation failed: client connection to 127.27.109.129:40963: connect: Connection refused (error 111)
I20260928 02:45:24.912230 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 6825
I20260928 02:45:24.918838 28086 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:45:25.262679 28086 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:45:25.262768 28086 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:45:25.606761 28086 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:45:25.606 UTC [6411] LOG: received smart shutdown request
2026-09-28 02:45:25.610 UTC [6411] LOG: background worker "logical replication launcher" (PID 6420) exited with exit code 1
2026-09-28 02:45:25.610 UTC [6415] LOG: shutting down
2026-09-28 02:45:25.763 UTC [6411] LOG: database system is shut down
2026-09-28T02:45:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44254 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:45:25 dist-test-slave-b5p5 krb5kdc[7394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:45:25 dist-test-slave-b5p5 krb5kdc[7394](info): set up 2 sockets
Sep 28 02:45:25 dist-test-slave-b5p5 krb5kdc[7394](info): commencing operation
krb5kdc: starting...
W20260928 02:45:26.906301 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:45:26 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:45:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangeradmin_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangeradmin_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangerlookup_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangerlookup_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
I20260928 02:45:27.056800 28086 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpfCjkR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/postgres -l logfile start
2026-09-28 02:45:29.307 UTC [7424] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:45:29.308 UTC [7424] LOG: listening on IPv4 address "127.27.109.148", port 55941
2026-09-28 02:45:29.311 UTC [7424] LOG: listening on Unix socket "/tmp/.s.PGSQL.55941"
2026-09-28 02:45:29.344 UTC [7427] LOG: database system was shut down at 2026-09-28 02:45:28 UTC
2026-09-28 02:45:29.348 UTC [7424] LOG: database system is ready to accept connections
I20260928 02:45:30.286295 28086 mini_postgres.cc:96] Postgres bound to 55941
2026-09-28 02:45:30.292 UTC [7435] FATAL: database "slave" does not exist
127.27.109.148:55941 - accepting connections
I20260928 02:45:30.292721 28086 mini_ranger.cc:161] Starting Ranger...
I20260928 02:45:30.306911 28086 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:45:30.694965 28086 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:45:30.695048 28086 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin
2026-09-28 02:45:30,964 [I] DB FLAVOR :POSTGRES
2026-09-28 02:45:30,965 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:45:30,965 [I] Checking connection..
2026-09-28 02:45:30,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:45:31,219 [I] Checking connection passed.
2026-09-28 02:45:31,219 [I] --------- Verifying version history table ---------
2026-09-28 02:45:31,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:45:31,464 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:45:31,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:45:31,700 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:45:31,701 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:45:31,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:45:31.940 UTC [7499] WARNING: there is no transaction in progress
2026-09-28 02:45:31,946 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:45:31,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:45:32,187 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:45:32,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:45:32,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:45:32,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:45:32,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b5p5.c.gcp-upstream.internal','N') ;"
2026-09-28 02:45:32,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:45:33,164 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:45:33,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:45:33,414 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:45:33,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:45:33,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:45:33,883 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:45:33,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:45:34.752 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.768 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.783 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.799 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.815 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.831 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.884 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.891 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.905 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.915 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.925 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.938 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.947 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.953 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.959 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:34.965 UTC [7607] WARNING: there is no transaction in progress
2026-09-28 02:45:35,544 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:45:35,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:45:35,789 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:45:35,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:45:36,023 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:45:36,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:45:36,273 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:45:36,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:45:36,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:45:36,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b5p5.c.gcp-upstream.internal';"
2026-09-28 02:45:36,997 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:45:36,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:45:37,232 [I] DB_PATCHES have already been applied
I20260928 02:45:37.238302 28086 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpfCjkR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:45:37.238361 28086 mini_ranger.cc:193] Using host: 127.27.109.148
I20260928 02:45:37.241338 28086 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.148:46401
Sep 28, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:45:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:45:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:45:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:45:38.336122 28086 mini_ranger.cc:160] Time spent starting Ranger: real 8.043s user 0.006s sys 0.002s
Sep 28, 2026 2:45:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:45:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:45:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:45:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:45:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:45:58.423779 28086 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangerkms_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/rangerkms_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/HTTP_127.27.109.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:45:58 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:45:58.500133 28086 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:45:58.508570 28086 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:45:59.305344 28086 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:45:59.305442 28086 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms
I20260928 02:45:59.305473 28086 mini_ranger_kms.cc:227] Using postgres at 127.27.109.148:55941
2026-09-28 02:45:59,466 [I] DB FLAVOR :POSTGRES
2026-09-28 02:45:59,466 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:45:59,466 [I] Checking connection
2026-09-28 02:45:59,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:45:59,692 [I] connection success
2026-09-28 02:45:59,693 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:45:59,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:45:59,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:45:59,915 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:45:59,915 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:45:59,915 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:45:59,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.148:55941/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:46:00,183 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:46:00.668026 28086 mini_ranger_kms.cc:310] Created kms service
I20260928 02:46:00.873615 28086 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:46:00.914181 28086 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:46:00 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563558, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:01.313759 28086 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:46:01.313956 28086 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpfCjkR/build/release/bin/postgresql.jar:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpfCjkR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:46:01.313977 28086 mini_ranger_kms.cc:240] Using host: 127.27.109.148
I20260928 02:46:01.317093 28086 mini_ranger_kms.cc:276] Ranger KMS PID: 7787
I20260928 02:46:01.317171 28086 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.148:41517
Sep 28, 2026 2:46:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:46:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:46:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:46:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:46:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:46:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:46:03.590580 28086 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.090s user 0.000s sys 0.143s
I20260928 02:46:03.590698 28086 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:46:03.590750 28086 mini_ranger_kms.cc:401] 127.27.109.148:41517/kms/v1/keys
Sep 28 02:46:03 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:46:04 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
Sep 28 02:46:04 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563563, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.148@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
Sep 28 02:46:06 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:46:06.618803 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37095
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37095
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:06.699764 7838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:06.699965 7838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:06.699986 7838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:06.701155 7838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:06.701197 7838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:06.701211 7838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:06.701248 7838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:06.701282 7838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:06.702802 7838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.148:41517/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37095
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:06.703049 7838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:06.703282 7838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:06.708542 7846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:06.708554 7844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:06.708698 7838 server_base.cc:1034] running on GCE node
W20260928 02:46:06.708630 7843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:06.709043 7838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:06.709298 7838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:06.710434 7838 hybrid_clock.cc:648] HybridClock initialized: now 1790563566710414 us; error 36 us; skew 500 ppm
Sep 28 02:46:06 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:06.715543 7838 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:06.715970 7838 webserver.cc:460] Webserver started at http://127.27.109.190:34677/ using document root <none> and password file <none>
I20260928 02:46:06.716163 7838 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:06.716198 7838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:06.716284 7838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:46:06 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:06.775949 7838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "97564e3e63194d0c849f9f331f386a2b"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-b5p5"
server_key: "e0d2eb775a822e5fa984025148922e5a"
server_key_iv: "6cacae6099d6e7e3b8204c415331a16c"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:06.776387 7838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "97564e3e63194d0c849f9f331f386a2b"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-b5p5"
server_key: "e0d2eb775a822e5fa984025148922e5a"
server_key_iv: "6cacae6099d6e7e3b8204c415331a16c"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:06.777819 7838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:46:06.791661 7853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:06.793053 7838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:06.793165 7838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "97564e3e63194d0c849f9f331f386a2b"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-b5p5"
server_key: "e0d2eb775a822e5fa984025148922e5a"
server_key_iv: "6cacae6099d6e7e3b8204c415331a16c"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:06.793273 7838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:06.822218 7838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:06.824882 7838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:06.825088 7838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:06.830376 7838 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37095
I20260928 02:46:06.830422 7905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37095 every 8 connection(s)
I20260928 02:46:06.830926 7838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:06.831697 7906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:06.834043 7906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Bootstrap starting.
I20260928 02:46:06.834749 7906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:06.835072 7906 log.cc:824] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:06.835923 7906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: No bootstrap required, opened a new log
I20260928 02:46:06.836596 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 7838
I20260928 02:46:06.836704 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:46:06.836871 28086 external_mini_cluster.cc:1428] Setting key caf8c15d70a8047583ae287b62b80470
I20260928 02:46:06.838660 7906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:06.838832 7906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:06.838883 7906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97564e3e63194d0c849f9f331f386a2b, State: Initialized, Role: FOLLOWER
I20260928 02:46:06.839028 7906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:06.839121 7906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:06.839154 7906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:06.839195 7906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:06.839920 7906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:06.840027 7906 leader_election.cc:302] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97564e3e63194d0c849f9f331f386a2b; no voters:
I20260928 02:46:06.840185 7906 leader_election.cc:288] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:06.840224 7911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:06.840373 7911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 LEADER]: Becoming Leader. State: Replica: 97564e3e63194d0c849f9f331f386a2b, State: Running, Role: LEADER
I20260928 02:46:06.840461 7911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:06.840533 7906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:06.840790 7911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97564e3e63194d0c849f9f331f386a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } } }
I20260928 02:46:06.840866 7911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:06.840991 7911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97564e3e63194d0c849f9f331f386a2b. Latest consensus state: current_term: 1 leader_uuid: "97564e3e63194d0c849f9f331f386a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } } }
I20260928 02:46:06.841042 7911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:06.842072 7906 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:46:06.846884 7915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:06.847636 7915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:46:06 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:06.852478 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:06.837373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.27.109.190:37095)
0928 02:46:06.837685 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:06.837695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:06.837727 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:46:06.838627 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:06.838636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:06.838702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:06.839959 (+ 1257us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:06.840156 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.841871 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.844037 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:06.845197 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:06.845333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:06.848559 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:06.848563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:06.848564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:06.848576 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:06.850766 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.851147 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.851150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.851151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.851267 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:06.851499 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:06.851500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:06.851501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:06.851823 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:06.851954 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:06.852055 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:46:06.852190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":69,"threads_started":1}
I20260928 02:46:06.855985 7915 catalog_manager.cc:1347] Generated new cluster ID: e780035125db45279a4c4239b4e3b6ba
I20260928 02:46:06.856030 7915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:06.878628 7915 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:06.879419 7915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:06.893216 7915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Generated new TSK 0
I20260928 02:46:06.893407 7915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:46:07.580801 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:07.679728 7951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:07.679984 7951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:07.680017 7951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:07.681612 7951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:07.681666 7951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:07.681704 7951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:07.683732 7951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:07.683954 7951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:07.684224 7951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:07.690182 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.690187 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.690493 7959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:07.691391 7951 server_base.cc:1034] running on GCE node
I20260928 02:46:07.691596 7951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:07.691843 7951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:07.693019 7951 hybrid_clock.cc:648] HybridClock initialized: now 1790563567693003 us; error 29 us; skew 500 ppm
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:07.695835 7951 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:07.696357 7951 webserver.cc:460] Webserver started at http://127.27.109.129:43555/ using document root <none> and password file <none>
I20260928 02:46:07.696609 7951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:07.696671 7951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:07.696851 7951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:07.710189 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "b825fb3ffd95408e8f95f915e781e216"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "166c33daaa9ac3995c2cf7abef4d6131"
server_key_iv: "8d99f811742557d89c1c7ae64b409f7d"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.710685 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "b825fb3ffd95408e8f95f915e781e216"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "166c33daaa9ac3995c2cf7abef4d6131"
server_key_iv: "8d99f811742557d89c1c7ae64b409f7d"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.712469 7951 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:07.734611 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.734870 7951 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 02:46:07.734965 7951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "b825fb3ffd95408e8f95f915e781e216"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "166c33daaa9ac3995c2cf7abef4d6131"
server_key_iv: "8d99f811742557d89c1c7ae64b409f7d"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.735049 7951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:07.758319 7951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:07.760802 7951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:07.760950 7951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:07.761206 7951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:07.761613 7951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:07.761663 7951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.761713 7951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:07.761739 7951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.769115 7951 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:40783
I20260928 02:46:07.769236 8081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:40783 every 8 connection(s)
I20260928 02:46:07.769534 7951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:07.776746 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 7951
I20260928 02:46:07.776856 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:07.777074 28086 external_mini_cluster.cc:1428] Setting key 3c4619f080b0e9b37606dd81c5674b1b
I20260928 02:46:07.780962 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.770293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:37719 (local address 127.27.109.190:37095)
0928 02:46:07.770553 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:07.770555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:07.770613 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:46:07.771307 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:07.771308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:07.771321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:07.771406 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:07.772113 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.772911 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.774391 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.774643 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.774700 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.776978 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:07.776980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:07.776980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:07.776987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:07.779450 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.779855 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.779856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.779857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.779919 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:07.780203 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:07.780204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:07.780204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:07.780350 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:07.780427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:07.780766 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 02:46:07.780860 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":88,"threads_started":1}
I20260928 02:46:07.781149 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:07.770300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.129:37719)
0928 02:46:07.770492 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:07.770539 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:07.771181 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:07.771499 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:07.771503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:07.771748 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:07.772014 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.772021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.773017 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.773018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.774298 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:07.774301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:07.774746 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:07.774747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:07.774885 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:07.775379 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:07.775389 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:07.776866 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:07.779549 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.779555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.779557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.779767 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.779986 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:07.779987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:07.779988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:07.780143 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:07.780428 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:07.780434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:07.780600 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:46:07.780713 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:46:07.780870 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260928 02:46:07.781751 8082 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:07.781852 8082 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:07.782047 8082 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:07.782615 7870 ts_manager.cc:194] Registered new tserver with Master: b825fb3ffd95408e8f95f915e781e216 (127.27.109.129:40783)
I20260928 02:46:07.783480 7870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:37719
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:46:07.802891 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:46:07.921038 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:07.921353 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:07.921388 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:07.923112 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:07.923182 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:07.923230 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:07.925372 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:07.925669 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:07.925933 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:07.932307 8098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.932272 8095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:07.932598 8096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:07.932885 8088 server_base.cc:1034] running on GCE node
I20260928 02:46:07.933069 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:07.933326 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:07.934540 8088 hybrid_clock.cc:648] HybridClock initialized: now 1790563567934513 us; error 39 us; skew 500 ppm
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:07.937527 8088 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:07.938030 8088 webserver.cc:460] Webserver started at http://127.27.109.130:44445/ using document root <none> and password file <none>
I20260928 02:46:07.938295 8088 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:07.938360 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:07.938521 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:46:07 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:07.950809 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "48f00ddb6733427b817b591e752b0ebc"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "8a51e15d2b84ec066659ed289fd6dfb1"
server_key_iv: "c4d23775fa5f168073c079e8026ce529"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.951224 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "48f00ddb6733427b817b591e752b0ebc"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "8a51e15d2b84ec066659ed289fd6dfb1"
server_key_iv: "c4d23775fa5f168073c079e8026ce529"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.955174 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:46:07.978612 8105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:07.978790 8088 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260928 02:46:07.978880 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "48f00ddb6733427b817b591e752b0ebc"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "8a51e15d2b84ec066659ed289fd6dfb1"
server_key_iv: "c4d23775fa5f168073c079e8026ce529"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:07.978974 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.001621 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.003929 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.004103 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.004310 8088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:08.004626 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:08.004680 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.004719 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:08.004740 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.009598 8088 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:35811
I20260928 02:46:08.009954 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:08.013006 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8088
I20260928 02:46:08.013067 8219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:35811 every 8 connection(s)
I20260928 02:46:08.013111 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:08.013316 28086 external_mini_cluster.cc:1428] Setting key a07bcb7701aec62c4c73c702b5fcf59b
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563567, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:08.025631 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.014213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:49887 (local address 127.27.109.190:37095)
0928 02:46:08.014308 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.014310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.014319 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:08.016307 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.016309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.016326 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.016358 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.017159 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.017951 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.019466 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.019773 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.019817 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.022430 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.022431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.022432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.022438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.024226 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.024581 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.024583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.024583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.024639 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.024910 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.024911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.024911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.025063 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.025127 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.025444 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.025529 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:46:08.025857 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.013638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.130:49887)
0928 02:46:08.013824 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.013868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.016168 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.016544 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.016548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.016780 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:08.017044 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.017049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.018066 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.018067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.019363 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.019366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.019920 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.019920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.020059 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.020641 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:08.020651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:08.022325 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:08.024300 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.024305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.024307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.024499 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.024727 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.024727 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.024728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.024838 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.025154 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:08.025161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:08.025301 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.025411 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.025550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260928 02:46:08.026278 8220 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:08.026376 8220 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.026597 8220 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:08.026965 7870 ts_manager.cc:194] Registered new tserver with Master: 48f00ddb6733427b817b591e752b0ebc (127.27.109.130:35811)
I20260928 02:46:08.027348 7870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:49887
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:46:08.056488 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:08.163695 8225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:08.163911 8225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:08.163950 8225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:08.165063 8225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:08.165129 8225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:08.165174 8225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:08.167063 8225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:08.167313 8225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:08.167563 8225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:08.172755 8231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.172806 8233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.172755 8230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.172924 8225 server_base.cc:1034] running on GCE node
I20260928 02:46:08.173202 8225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:08.173446 8225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:08.174610 8225 hybrid_clock.cc:648] HybridClock initialized: now 1790563568174587 us; error 41 us; skew 500 ppm
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:08.177181 8225 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:08.177606 8225 webserver.cc:460] Webserver started at http://127.27.109.131:34117/ using document root <none> and password file <none>
I20260928 02:46:08.177843 8225 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:08.177910 8225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:08.178058 8225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:08.191704 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "74b2758eb7fd44f4a251cc1250c21c6c"
format_stamp: "Formatted at 2026-09-28 02:46:08 on dist-test-slave-b5p5"
server_key: "a38e4d13cc539fba783e9e227dcf035a"
server_key_iv: "7e3b165b477f7380f844c752dd41ddba"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.192133 8225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "74b2758eb7fd44f4a251cc1250c21c6c"
format_stamp: "Formatted at 2026-09-28 02:46:08 on dist-test-slave-b5p5"
server_key: "a38e4d13cc539fba783e9e227dcf035a"
server_key_iv: "7e3b165b477f7380f844c752dd41ddba"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.193783 8225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:46:08.209012 8240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.209231 8225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:08.209322 8225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "74b2758eb7fd44f4a251cc1250c21c6c"
format_stamp: "Formatted at 2026-09-28 02:46:08 on dist-test-slave-b5p5"
server_key: "a38e4d13cc539fba783e9e227dcf035a"
server_key_iv: "7e3b165b477f7380f844c752dd41ddba"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.209407 8225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.231710 8225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.234031 8225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.234193 8225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.234454 8225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:08.234864 8225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:08.234915 8225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.234951 8225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:08.234970 8225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.242169 8225 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:41351
I20260928 02:46:08.242663 8225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:08.242969 8353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:41351 every 8 connection(s)
I20260928 02:46:08.243834 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8225
I20260928 02:46:08.244035 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:08.244278 28086 external_mini_cluster.cc:1428] Setting key 89a46739e679b5905214b40857e52970
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:08.254253 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.244450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:55051 (local address 127.27.109.190:37095)
0928 02:46:08.244496 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.244498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.244509 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:08.245001 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.245002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.245014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.245057 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.245769 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.246556 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.248045 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.248366 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.248410 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.250997 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.250999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.250999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.251006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.252691 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.253173 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.253174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.253174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.253248 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.253529 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.253529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.253530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.253659 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.253701 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.254062 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.254145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:46:08.255015 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.243845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.131:55051)
0928 02:46:08.244045 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.244150 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.244870 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.245129 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.245133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.245381 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:08.245653 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.245660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.246671 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.246672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.247940 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.247943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.248513 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.248514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.248654 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.249250 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:08.249260 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:08.250888 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:08.252787 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.252793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.252795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.253079 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.253325 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.253325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.253326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.253452 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.253748 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:08.253753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:08.253906 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.254623 (+ 717us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.254784 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260928 02:46:08.255671 8354 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:08.255761 8354 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.255939 8354 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:08.256358 7870 ts_manager.cc:194] Registered new tserver with Master: 74b2758eb7fd44f4a251cc1250c21c6c (127.27.109.131:41351)
I20260928 02:46:08.256834 7870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:55051
I20260928 02:46:08.257582 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:08.258126 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 7951
I20260928 02:46:08.264909 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8088
I20260928 02:46:08.271782 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8225
I20260928 02:46:08.277104 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 7838
I20260928 02:46:08.282339 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37095
--webserver_interface=127.27.109.190
--webserver_port=34677
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37095
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:08.361991 8357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:08.362212 8357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:08.362249 8357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:08.363420 8357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:08.363485 8357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:08.363510 8357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:08.363540 8357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:08.363566 8357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:08.365060 8357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37095
--ranger_config_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:08.365317 8357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:08.365522 8357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:08.370302 8365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.370291 8363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.370293 8362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.370498 8357 server_base.cc:1034] running on GCE node
I20260928 02:46:08.370708 8357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:08.370980 8357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:08.372152 8357 hybrid_clock.cc:648] HybridClock initialized: now 1790563568372137 us; error 45 us; skew 500 ppm
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:08.374619 8357 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:08.375058 8357 webserver.cc:460] Webserver started at http://127.27.109.190:34677/ using document root <none> and password file <none>
I20260928 02:46:08.375321 8357 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:08.375373 8357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:08.426744 8357 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.427553 8373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.427757 8357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.427855 8357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "97564e3e63194d0c849f9f331f386a2b"
format_stamp: "Formatted at 2026-09-28 02:46:06 on dist-test-slave-b5p5"
server_key: "e0d2eb775a822e5fa984025148922e5a"
server_key_iv: "6cacae6099d6e7e3b8204c415331a16c"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.428226 8357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.436394 8357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.438037 8357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.438141 8357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.441978 8357 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37095
I20260928 02:46:08.442070 8425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37095 every 8 connection(s)
I20260928 02:46:08.442384 8357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:08.443594 8426 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:08.444250 8426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Bootstrap starting.
I20260928 02:46:08.447414 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8357
I20260928 02:46:08.447700 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:40783
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:37095
--webserver_port=43555
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:08.452117 8426 log.cc:824] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:08.453665 8426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:08.453948 8426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Bootstrap complete.
I20260928 02:46:08.456281 8426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:08.456543 8426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97564e3e63194d0c849f9f331f386a2b, State: Initialized, Role: FOLLOWER
I20260928 02:46:08.456746 8426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:08.456847 8426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:08.456899 8426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:08.456945 8426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:08.457867 8426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:08.457983 8426 leader_election.cc:302] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97564e3e63194d0c849f9f331f386a2b; no voters:
I20260928 02:46:08.458178 8426 leader_election.cc:288] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:46:08.458218 8430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:46:08.458321 8430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [term 2 LEADER]: Becoming Leader. State: Replica: 97564e3e63194d0c849f9f331f386a2b, State: Running, Role: LEADER
I20260928 02:46:08.458400 8430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } }
I20260928 02:46:08.458426 8426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:08.458704 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97564e3e63194d0c849f9f331f386a2b. Latest consensus state: current_term: 2 leader_uuid: "97564e3e63194d0c849f9f331f386a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } } }
I20260928 02:46:08.458804 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:08.458977 8434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:08.459471 8434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:08.459596 8434 catalog_manager.cc:1259] Loaded cluster ID: e780035125db45279a4c4239b4e3b6ba
I20260928 02:46:08.459633 8434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:08.459993 8426 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:46:08.460403 8431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "97564e3e63194d0c849f9f331f386a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97564e3e63194d0c849f9f331f386a2b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37095 } } }
I20260928 02:46:08.460484 8431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:08.460812 8434 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:08.463404 8434 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Loaded TSK: 0
I20260928 02:46:08.463873 8434 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:46:08.552673 8428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:08.552894 8428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:08.552933 8428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:08.554176 8428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:08.554239 8428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:08.554281 8428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:08.555756 8428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:40783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=43555
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:08.556018 8428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:08.556236 8428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:08.564829 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.564877 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.565143 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.565492 8428 server_base.cc:1034] running on GCE node
I20260928 02:46:08.565694 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:08.565914 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:08.567068 8428 hybrid_clock.cc:648] HybridClock initialized: now 1790563568567039 us; error 46 us; skew 500 ppm
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:08.569948 8428 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:08.570451 8428 webserver.cc:460] Webserver started at http://127.27.109.129:43555/ using document root <none> and password file <none>
I20260928 02:46:08.570711 8428 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:08.570794 8428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:08.591233 8428 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:46:08.592224 8474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.592382 8428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.592463 8428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "b825fb3ffd95408e8f95f915e781e216"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "166c33daaa9ac3995c2cf7abef4d6131"
server_key_iv: "8d99f811742557d89c1c7ae64b409f7d"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.592924 8428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.617628 8428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.619896 8428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.620074 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.620312 8428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:08.620693 8428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:08.620756 8428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.620798 8428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:08.620824 8428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.627926 8428 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:40783
I20260928 02:46:08.628309 8428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:08.628660 8587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:40783 every 8 connection(s)
I20260928 02:46:08.634660 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8428
I20260928 02:46:08.635041 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:35811
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:37095
--webserver_port=44445
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:08.640070 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.629236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.129:59609)
0928 02:46:08.629550 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.629597 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.630225 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.630764 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.630768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.630979 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:08.631241 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.631247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.632474 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.632476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.633703 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.633706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.634264 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.634265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.634398 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.634992 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:08.635001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:08.636470 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:08.638214 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.638220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.638222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.638536 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.638831 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.638832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.638833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.638961 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.639416 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:08.639422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:08.639569 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.639702 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.639849 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":82,"threads_started":1}
I20260928 02:46:08.640142 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.629327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:59609 (local address 127.27.109.190:37095)
0928 02:46:08.629542 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.629550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.629700 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:46:08.630489 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.630496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.630555 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.630697 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.631365 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.632373 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.633786 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.634158 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.634233 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.636581 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.636584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.636584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.636593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.638126 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.638626 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.638629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.638630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.638740 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.639033 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.639034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.639034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.639333 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.639418 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.639770 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.639900 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260928 02:46:08.640882 8588 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:08.640976 8588 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.641150 8588 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:08.642069 8588 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37095 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:46:08.642141 8588 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.642203 8588 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:08.642597 8588 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.642688 8588 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:08.643041 8588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:46:08.737846 8592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:08.738094 8592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:08.738131 8592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:08.739754 8592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:08.739802 8592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:08.739840 8592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:08.741725 8592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:35811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=44445
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:08.741964 8592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:08.742197 8592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:08.747290 8597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.747565 8598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.748368 8592 server_base.cc:1034] running on GCE node
W20260928 02:46:08.750594 8600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.750900 8592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:08.751159 8592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:08.754423 8592 hybrid_clock.cc:648] HybridClock initialized: now 1790563568754405 us; error 27 us; skew 500 ppm
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:08.757190 8592 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:08.757747 8592 webserver.cc:460] Webserver started at http://127.27.109.130:44445/ using document root <none> and password file <none>
I20260928 02:46:08.757977 8592 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:08.758059 8592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:08.776609 8592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.777552 8608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.777737 8592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.777827 8592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "48f00ddb6733427b817b591e752b0ebc"
format_stamp: "Formatted at 2026-09-28 02:46:07 on dist-test-slave-b5p5"
server_key: "8a51e15d2b84ec066659ed289fd6dfb1"
server_key_iv: "c4d23775fa5f168073c079e8026ce529"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.778148 8592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.791620 8592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.793943 8592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.794093 8592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.794333 8592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:08.794714 8592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:08.794765 8592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.794796 8592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:08.794821 8592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.802193 8592 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:35811
I20260928 02:46:08.802685 8592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:08.803185 8721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:35811 every 8 connection(s)
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:08.811713 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.803569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:55765 (local address 127.27.109.190:37095)
0928 02:46:08.803613 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.803615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.803864 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:46:08.804413 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.804415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.804430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.804482 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.805026 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.805524 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.806540 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.806768 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.806856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.808772 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.808774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.808775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.808782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.810331 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.810770 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.810771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.810772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.810830 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.811092 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.811093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.811094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.811208 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:08.811280 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:08.811545 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:46:08.811626 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:46:08.811887 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.803572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.130:55765)
0928 02:46:08.803738 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.803772 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.804304 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.804561 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.804565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.804734 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:08.804916 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.804920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.805629 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.805630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.806405 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.806407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.806883 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.806884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.807009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.807494 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:08.807502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:08.808680 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:08.810429 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.810435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.810437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.810685 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.810911 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.810912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.810912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.811023 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.811281 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:08.811288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:08.811418 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:46:08.811524 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:46:08.811666 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 02:46:08.812630 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8592
I20260928 02:46:08.812939 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.148:41517/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:41351
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:37095
--webserver_port=34117
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:08.814436 8722 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:08.814560 8722 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.814755 8722 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:08.815502 8722 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37095 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:46:08.815578 8722 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.815641 8722 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:08.815939 8722 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:08.816020 8722 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:08.816275 8722 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:46:08.921809 8725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:08.922068 8725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:08.922104 8725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:08.923802 8725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:08.923853 8725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:08.923892 8725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:08.925957 8725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.148:41517/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:41351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=34117
--tserver_master_addrs=127.27.109.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:08.926172 8725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:08.926401 8725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:08.932773 8733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.932796 8730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:08.932775 8731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:08.933154 8725 server_base.cc:1034] running on GCE node
I20260928 02:46:08.933353 8725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:08.933636 8725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:08.934803 8725 hybrid_clock.cc:648] HybridClock initialized: now 1790563568934775 us; error 43 us; skew 500 ppm
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:08.937930 8725 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:08.938413 8725 webserver.cc:460] Webserver started at http://127.27.109.131:34117/ using document root <none> and password file <none>
I20260928 02:46:08.938687 8725 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:08.938761 8725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:08.956751 8725 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:08.957748 8742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.957904 8725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:08.957984 8725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "74b2758eb7fd44f4a251cc1250c21c6c"
format_stamp: "Formatted at 2026-09-28 02:46:08 on dist-test-slave-b5p5"
server_key: "a38e4d13cc539fba783e9e227dcf035a"
server_key_iv: "7e3b165b477f7380f844c752dd41ddba"
server_key_version: "kuduclusterkey@0"
I20260928 02:46:08.958501 8725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:08.978626 8725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:08.980902 8725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:08.981060 8725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:08.981314 8725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:08.981693 8725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:08.981742 8725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.981773 8725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:08.981791 8725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:08.989120 8725 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:41351
I20260928 02:46:08.989200 8855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:41351 every 8 connection(s)
I20260928 02:46:08.989563 8725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:08.992702 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8725
Sep 28 02:46:08 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563568, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:09.000765 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.990342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:45031 (local address 127.27.109.190:37095)
0928 02:46:08.990391 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.990393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.990650 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:46:08.991344 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.991346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.991359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.991391 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.992201 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.992988 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.994837 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.995145 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.995187 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.997528 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.997529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.997529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.997536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:08.999179 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:08.999649 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:08.999650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:08.999651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:08.999706 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.000003 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.000004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.000004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.000138 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:09.000179 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:09.000586 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 02:46:09.000655 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:46:09.001509 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.990327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37095 (local address 127.27.109.131:45031)
0928 02:46:08.990520 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:08.990564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:08.991200 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:08.991572 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:08.991576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:08.991790 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:08.992086 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.992092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.993468 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.993469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.994733 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:08.994736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.995295 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.995309 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:08.995438 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.995955 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:08.995965 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:08.997421 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:08.999256 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.999262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.999265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.999530 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:08.999781 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:08.999782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:08.999782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:08.999913 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:09.000250 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:09.000256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:09.000417 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 02:46:09.001104 (+ 687us) client_negotiation.cc:240] Negotiation successful
0928 02:46:09.001250 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260928 02:46:09.002148 8856 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37095
I20260928 02:46:09.002204 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:08.993238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.27.109.190:37095)
0928 02:46:08.994911 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:08.994918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:08.994925 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:08.994943 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:08.994943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:08.994952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:08.995035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:08.995625 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.996360 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.997476 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:08.997793 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:08.997850 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:08.999153 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:08.999155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:08.999155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:08.999161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:09.000993 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.001383 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.001384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.001384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.001437 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:09.001641 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:09.001643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:09.001643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:09.001757 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:09.001944 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:09.001995 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:46:09.002081 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":50,"threads_started":1}
I20260928 02:46:09.002233 8856 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.002416 8856 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:09.003260 8856 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37095 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:46:09.003345 8856 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.003412 8856 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:09.003680 8856 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.003752 8856 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
W20260928 02:46:09.004040 8856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:46:09 dist-test-slave-b5p5 krb5kdc[7394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:09.643222 8588 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.643402 8588 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:09.643779 8390 ts_manager.cc:194] Registered new tserver with Master: b825fb3ffd95408e8f95f915e781e216 (127.27.109.129:40783)
I20260928 02:46:09.644482 8390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:59609
I20260928 02:46:09.816458 8722 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:09.816624 8722 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:09.816953 8390 ts_manager.cc:194] Registered new tserver with Master: 48f00ddb6733427b817b591e752b0ebc (127.27.109.130:35811)
I20260928 02:46:09.817505 8390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:55765
Sep 28 02:46:09 dist-test-slave-b5p5 krb5kdc[7394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563569, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for HTTP/127.27.109.148@KRBTEST.COM
I20260928 02:46:10.004230 8856 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:10.004361 8856 heartbeater.cc:511] Master 127.27.109.190:37095 requested a full tablet report, sending...
I20260928 02:46:10.004653 8390 ts_manager.cc:194] Registered new tserver with Master: 74b2758eb7fd44f4a251cc1250c21c6c (127.27.109.131:41351)
I20260928 02:46:10.005162 8390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:45031
I20260928 02:46:10.009806 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:10.017021 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.010798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.27.109.190:37095)
0928 02:46:10.010930 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.010932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.010941 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:10.010990 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.010991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.011002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.011063 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:10.011618 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.012101 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.012819 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.013036 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.013110 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.014389 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:10.014390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:10.014391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:10.014398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:10.015854 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.016243 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.016244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.016244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.016305 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:10.016510 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:10.016511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:10.016512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:10.016627 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:10.016735 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.016832 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.016915 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 02:46:10.018894 8390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:10.020735 8390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:10.027824 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.023762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.27.109.131:41351)
0928 02:46:10.024696 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.024700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.024710 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:46:10.024740 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.024742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.024756 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.024853 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.024955 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.025664 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.026857 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.027372 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.027489 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.027552 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.027706 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.027746 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":66,"threads_started":1}
I20260928 02:46:10.027822 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.023642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:41351 (local address 127.0.0.1:37962)
0928 02:46:10.023996 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.024036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.024102 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.024849 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.024850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.024852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.024898 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.024900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.025751 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.025752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.026773 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.026775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.027490 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.027491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.027577 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.027590 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.027688 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.027745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260928 02:46:10.028527 8782 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4fdd61d8194062bc7ca266243d316a (DEFAULT_TABLE table=test-table [id=4f01d489d11a4160bfed219e389c1a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.028900 8782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4fdd61d8194062bc7ca266243d316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.029685 8886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.023642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40783 (local address 127.0.0.1:42010)
0928 02:46:10.023996 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.024039 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.024102 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.024556 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.024559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.024562 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.024626 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.024628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.026511 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.026512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.028422 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.028425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.029469 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.029470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.029530 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.029534 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.029585 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.029624 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":75,"threads_started":1}
I20260928 02:46:10.029735 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.023891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.27.109.129:40783)
0928 02:46:10.024323 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.024327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.024340 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:46:10.024400 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.024402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.024418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.024490 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.025096 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.026400 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.028522 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.029314 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.029364 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.029409 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.029619 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.029659 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":50,"threads_started":1}
I20260928 02:46:10.030339 8522 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4fdd61d8194062bc7ca266243d316a (DEFAULT_TABLE table=test-table [id=4f01d489d11a4160bfed219e389c1a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.030650 8522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4fdd61d8194062bc7ca266243d316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.032199 8889 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Bootstrap starting.
I20260928 02:46:10.032912 8889 tablet_bootstrap.cc:654] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.033233 8889 log.cc:824] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.033951 8891 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: Bootstrap starting.
I20260928 02:46:10.034614 8891 tablet_bootstrap.cc:654] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.034902 8891 log.cc:824] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.035570 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.023854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35811 (local address 127.0.0.1:58506)
0928 02:46:10.027997 (+ 4143us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.028009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.028077 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.028281 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.028282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.028284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.028315 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.028317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.029315 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.029316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.034404 (+ 5088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.034407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.035337 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.035339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.035407 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.035411 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.035471 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.035506 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4132,"thread_start_us":68,"threads_started":1}
I20260928 02:46:10.035634 8889 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: No bootstrap required, opened a new log
I20260928 02:46:10.035696 8889 ts_tablet_manager.cc:1397] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:46:10.035919 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.024098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.27.109.130:35811)
0928 02:46:10.024609 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.024613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.028051 (+ 3438us) server_negotiation.cc:407] Connection header received
0928 02:46:10.028167 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.028169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.028187 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.028286 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.028369 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.029237 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.034521 (+ 5284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.035262 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.035337 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.035382 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.035806 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.035849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":56,"threads_started":1}
I20260928 02:46:10.036698 8645 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4fdd61d8194062bc7ca266243d316a (DEFAULT_TABLE table=test-table [id=4f01d489d11a4160bfed219e389c1a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:10.037025 8645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4fdd61d8194062bc7ca266243d316a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:10.039170 8889 raft_consensus.cc:348] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.039377 8889 raft_consensus.cc:374] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.039418 8889 raft_consensus.cc:729] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b2758eb7fd44f4a251cc1250c21c6c, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.039561 8889 consensus_queue.cc:260] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.039798 8889 ts_tablet_manager.cc:1428] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 02:46:10.039906 8856 heartbeater.cc:503] Master 127.27.109.190:37095 was elected leader, sending a full tablet report...
I20260928 02:46:10.040741 8894 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Bootstrap starting.
I20260928 02:46:10.041147 8891 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: No bootstrap required, opened a new log
I20260928 02:46:10.041205 8891 ts_tablet_manager.cc:1397] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 02:46:10.041337 8894 tablet_bootstrap.cc:654] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:10.041603 8894 log.cc:824] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:10.042723 8894 tablet_bootstrap.cc:492] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: No bootstrap required, opened a new log
I20260928 02:46:10.042779 8894 ts_tablet_manager.cc:1397] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:10.043514 8891 raft_consensus.cc:348] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.043679 8891 raft_consensus.cc:374] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.043735 8891 raft_consensus.cc:729] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b825fb3ffd95408e8f95f915e781e216, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.043867 8891 consensus_queue.cc:260] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.044090 8891 ts_tablet_manager.cc:1428] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:46:10.044190 8588 heartbeater.cc:503] Master 127.27.109.190:37095 was elected leader, sending a full tablet report...
I20260928 02:46:10.044929 8894 raft_consensus.cc:348] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.045073 8894 raft_consensus.cc:374] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:10.045112 8894 raft_consensus.cc:729] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f00ddb6733427b817b591e752b0ebc, State: Initialized, Role: FOLLOWER
I20260928 02:46:10.045244 8894 consensus_queue.cc:260] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.045485 8894 ts_tablet_manager.cc:1428] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:46:10.045496 8722 heartbeater.cc:503] Master 127.27.109.190:37095 was elected leader, sending a full tablet report...
I20260928 02:46:10.046156 8896 raft_consensus.cc:482] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:10.046234 8896 raft_consensus.cc:504] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.047022 8896 leader_election.cc:288] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74b2758eb7fd44f4a251cc1250c21c6c (127.27.109.131:41351), 48f00ddb6733427b817b591e752b0ebc (127.27.109.130:35811)
I20260928 02:46:10.050804 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.047308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:54993 (local address 127.27.109.130:35811)
0928 02:46:10.047343 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.047344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.047477 (+ 133us) server_negotiation.cc:407] Connection header received
0928 02:46:10.047566 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.047567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.047576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.047605 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.047789 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.048443 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.049830 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.050379 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.050423 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.050429 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.050719 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.050753 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:46:10.050804 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.046910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:45695 (local address 127.27.109.131:41351)
0928 02:46:10.046944 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.046946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.047202 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:46:10.047396 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.047397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.047408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.047461 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:10.047643 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.048360 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.049830 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.050376 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.050423 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.050429 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.050709 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.050745 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:46:10.051071 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.047124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35811 (local address 127.27.109.129:54993)
0928 02:46:10.047430 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.047438 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.047498 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.047660 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.047661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.047662 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.047707 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.047718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.048523 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.048523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.049728 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.049730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.050542 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.050543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.050623 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.050626 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.050942 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.050989 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260928 02:46:10.051353 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.046748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:41351 (local address 127.27.109.129:45695)
0928 02:46:10.047127 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.047139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:10.047230 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:10.047504 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:10.047505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:10.047506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:10.047567 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.047568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.048447 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.048448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.049727 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:10.049730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.050542 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.050543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:10.050623 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.050627 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:10.051085 (+ 458us) client_negotiation.cc:240] Negotiation successful
0928 02:46:10.051277 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":66,"thread_start_us":68,"threads_started":1}
I20260928 02:46:10.051373 8665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4fdd61d8194062bc7ca266243d316a" candidate_uuid: "b825fb3ffd95408e8f95f915e781e216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f00ddb6733427b817b591e752b0ebc" is_pre_election: true
I20260928 02:46:10.051486 8665 raft_consensus.cc:2454] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b825fb3ffd95408e8f95f915e781e216 in term 0.
I20260928 02:46:10.051748 8475 leader_election.cc:302] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f00ddb6733427b817b591e752b0ebc, b825fb3ffd95408e8f95f915e781e216; no voters:
I20260928 02:46:10.051733 8810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4fdd61d8194062bc7ca266243d316a" candidate_uuid: "b825fb3ffd95408e8f95f915e781e216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" is_pre_election: true
I20260928 02:46:10.051855 8810 raft_consensus.cc:2454] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b825fb3ffd95408e8f95f915e781e216 in term 0.
I20260928 02:46:10.051856 8896 raft_consensus.cc:2781] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:10.051914 8896 raft_consensus.cc:482] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:10.051945 8896 raft_consensus.cc:3037] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.052506 8896 raft_consensus.cc:504] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
I20260928 02:46:10.052634 8896 leader_election.cc:288] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [CANDIDATE]: Term 1 election: Requested vote from peers 74b2758eb7fd44f4a251cc1250c21c6c (127.27.109.131:41351), 48f00ddb6733427b817b591e752b0ebc (127.27.109.130:35811)
I20260928 02:46:10.052796 8665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4fdd61d8194062bc7ca266243d316a" candidate_uuid: "b825fb3ffd95408e8f95f915e781e216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f00ddb6733427b817b591e752b0ebc"
I20260928 02:46:10.052807 8810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4fdd61d8194062bc7ca266243d316a" candidate_uuid: "b825fb3ffd95408e8f95f915e781e216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b2758eb7fd44f4a251cc1250c21c6c"
I20260928 02:46:10.052889 8810 raft_consensus.cc:3037] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.052891 8665 raft_consensus.cc:3037] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:10.053495 8665 raft_consensus.cc:2454] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b825fb3ffd95408e8f95f915e781e216 in term 1.
I20260928 02:46:10.053637 8810 raft_consensus.cc:2454] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b825fb3ffd95408e8f95f915e781e216 in term 1.
I20260928 02:46:10.053654 8475 leader_election.cc:302] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f00ddb6733427b817b591e752b0ebc, b825fb3ffd95408e8f95f915e781e216; no voters:
I20260928 02:46:10.053733 8896 raft_consensus.cc:2781] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:10.053851 8896 raft_consensus.cc:686] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [term 1 LEADER]: Becoming Leader. State: Replica: b825fb3ffd95408e8f95f915e781e216, State: Running, Role: LEADER
I20260928 02:46:10.053929 8896 consensus_queue.cc:237] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } }
W20260928 02:46:10.054116 8723 tablet.cc:2369] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:10.054607 8390 catalog_manager.cc:5495] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 reported cstate change: term changed from 0 to 1, leader changed from <none> to b825fb3ffd95408e8f95f915e781e216 (127.27.109.129). New cstate: current_term: 1 leader_uuid: "b825fb3ffd95408e8f95f915e781e216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b825fb3ffd95408e8f95f915e781e216" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 40783 } health_report { overall_health: HEALTHY } } }
I20260928 02:46:10.063495 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.060689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.27.109.129:40783)
0928 02:46:10.060777 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:10.060779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:10.060786 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:10.060812 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:10.060813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:10.060829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:10.060904 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:10.061037 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.061479 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.062404 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:10.062680 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:10.062874 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:10.062938 (+ 64us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:10.063045 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:10.063119 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:10.063321 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 02:46:10.063344 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:46:10.065083 8665 raft_consensus.cc:1261] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 1 FOLLOWER]: Refusing update from remote peer b825fb3ffd95408e8f95f915e781e216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.065150 8810 raft_consensus.cc:1261] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 1 FOLLOWER]: Refusing update from remote peer b825fb3ffd95408e8f95f915e781e216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:10.065403 8896 consensus_queue.cc:1035] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b2758eb7fd44f4a251cc1250c21c6c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.065508 8900 consensus_queue.cc:1035] T 5d4fdd61d8194062bc7ca266243d316a P b825fb3ffd95408e8f95f915e781e216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f00ddb6733427b817b591e752b0ebc" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:10.071440 8903 mvcc.cc:205] Tried to move back new op lower bound from 7334148382984454144 to 7334148382941278208. Current Snapshot: MvccSnapshot[applied={T|T < 7334148382984454144}]
I20260928 02:46:10.071921 8904 mvcc.cc:205] Tried to move back new op lower bound from 7334148382984454144 to 7334148382941278208. Current Snapshot: MvccSnapshot[applied={T|T < 7334148382984454144}]
I20260928 02:46:10.073117 8905 mvcc.cc:205] Tried to move back new op lower bound from 7334148382984454144 to 7334148382941278208. Current Snapshot: MvccSnapshot[applied={T|T < 7334148382984454144}]
I20260928 02:46:10.074379 8390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:10.074484 8390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:10.075201 8390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97564e3e63194d0c849f9f331f386a2b: Sending DeleteTablet for 3 replicas of tablet 5d4fdd61d8194062bc7ca266243d316a
I20260928 02:46:10.075599 8522 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4fdd61d8194062bc7ca266243d316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:42010
I20260928 02:46:10.075649 8782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4fdd61d8194062bc7ca266243d316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:37962
I20260928 02:46:10.075706 8645 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d4fdd61d8194062bc7ca266243d316a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:10 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:58506
I20260928 02:46:10.076267 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8428
I20260928 02:46:10.078606 8914 tablet_replica.cc:330] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: stopping tablet replica
I20260928 02:46:10.078744 8915 tablet_replica.cc:330] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: stopping tablet replica
I20260928 02:46:10.078889 8914 raft_consensus.cc:2229] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:10.079015 8915 raft_consensus.cc:2229] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:10.079185 8914 raft_consensus.cc:2258] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:10.079347 8915 raft_consensus.cc:2258] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:10.079905 8915 ts_tablet_manager.cc:1905] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:10.081357 8914 ts_tablet_manager.cc:1905] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:10.081591 8915 ts_tablet_manager.cc:1918] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:10.081745 8915 log.cc:1196] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/5d4fdd61d8194062bc7ca266243d316a
I20260928 02:46:10.082120 8915 ts_tablet_manager.cc:1939] T 5d4fdd61d8194062bc7ca266243d316a P 48f00ddb6733427b817b591e752b0ebc: Deleting consensus metadata
I20260928 02:46:10.082592 8374 catalog_manager.cc:4841] TS 48f00ddb6733427b817b591e752b0ebc (127.27.109.130:35811): tablet 5d4fdd61d8194062bc7ca266243d316a (table test-table [id=4f01d489d11a4160bfed219e389c1a1e]) successfully deleted
I20260928 02:46:10.082985 8914 ts_tablet_manager.cc:1918] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:10.083070 8914 log.cc:1196] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/5d4fdd61d8194062bc7ca266243d316a
I20260928 02:46:10.083297 8914 ts_tablet_manager.cc:1939] T 5d4fdd61d8194062bc7ca266243d316a P 74b2758eb7fd44f4a251cc1250c21c6c: Deleting consensus metadata
I20260928 02:46:10.083652 8374 catalog_manager.cc:4841] TS 74b2758eb7fd44f4a251cc1250c21c6c (127.27.109.131:41351): tablet 5d4fdd61d8194062bc7ca266243d316a (table test-table [id=4f01d489d11a4160bfed219e389c1a1e]) successfully deleted
W20260928 02:46:10.086102 8375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:40783 (error 108)
I20260928 02:46:10.086594 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:10.086495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:40783 (local address 127.0.0.1:42034)
0928 02:46:10.086530 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:10.086563 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:40783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 02:46:10.086776 8375 catalog_manager.cc:4584] TS b825fb3ffd95408e8f95f915e781e216 (127.27.109.129:40783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d4fdd61d8194062bc7ca266243d316a: Network error: Client connection negotiation failed: client connection to 127.27.109.129:40783: connect: Connection refused (error 111)
I20260928 02:46:10.088169 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8592
I20260928 02:46:10.093546 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8725
I20260928 02:46:10.099408 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8357
I20260928 02:46:10.108357 28086 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:46:10.442104 28086 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:46:10.442188 28086 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:46:10.785996 28086 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:46:10.786 UTC [7424] LOG: received smart shutdown request
2026-09-28 02:46:10.789 UTC [7424] LOG: background worker "logical replication launcher" (PID 7433) exited with exit code 1
2026-09-28 02:46:10.789 UTC [7428] LOG: shutting down
2026-09-28 02:46:10.928 UTC [7424] LOG: database system is shut down
2026-09-28T02:46:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45170 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:11 dist-test-slave-b5p5 krb5kdc[8925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:11 dist-test-slave-b5p5 krb5kdc[8925](info): set up 2 sockets
Sep 28 02:46:11 dist-test-slave-b5p5 krb5kdc[8925](info): commencing operation
krb5kdc: starting...
W20260928 02:46:12.070384 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:46:12.178969 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:38605
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:12.256865 8938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:12.257050 8938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:12.257072 8938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:12.258104 8938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:12.258138 8938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:12.258152 8938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:12.258164 8938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:12.258177 8938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:12.259572 8938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:12.259742 8938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:12.259905 8938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:12.264578 8943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:12.264569 8946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:12.264585 8938 server_base.cc:1034] running on GCE node
W20260928 02:46:12.264569 8944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:12.265049 8938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:12.265282 8938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:12.266424 8938 hybrid_clock.cc:648] HybridClock initialized: now 1790563572266400 us; error 48 us; skew 500 ppm
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:12.271670 8938 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:12.272032 8938 webserver.cc:460] Webserver started at http://127.27.109.190:46687/ using document root <none> and password file <none>
I20260928 02:46:12.272217 8938 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:12.272257 8938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:12.272342 8938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:12.273350 8938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "ca344a8119bf4147b7ffd16868691ab6"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "cbfc8ee18408922e857a397fc114912f"
server_key_iv: "0329bf36dda1b0454a916ed39a3d4c3d"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.273635 8938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "ca344a8119bf4147b7ffd16868691ab6"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "cbfc8ee18408922e857a397fc114912f"
server_key_iv: "0329bf36dda1b0454a916ed39a3d4c3d"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.274902 8938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:46:12.275571 8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:12.275765 8938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:46:12.275862 8938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "ca344a8119bf4147b7ffd16868691ab6"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "cbfc8ee18408922e857a397fc114912f"
server_key_iv: "0329bf36dda1b0454a916ed39a3d4c3d"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.275914 8938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:12.292603 8938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:12.294906 8938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:12.295015 8938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:12.298681 8938 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:38605
I20260928 02:46:12.298760 9005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:38605 every 8 connection(s)
I20260928 02:46:12.299091 8938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:12.299902 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.301442 9006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.302906 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 8938
I20260928 02:46:12.303036 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:46:12.303225 28086 external_mini_cluster.cc:1428] Setting key e1d6a4cbae22b804af501355eb3ebb05
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.189@KRBTEST.COM; defaulting to no policy
I20260928 02:46:12.310979 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.302258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:38605 (local address 127.0.0.1:37684)
0928 02:46:12.302435 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.302512 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.302945 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.303273 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.303276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.303543 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.303689 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.303692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.304704 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.304705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.305360 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.305362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.305879 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.305880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.305943 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.306502 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.306511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.307830 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.309513 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.309519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.309521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.309705 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.309951 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.309952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.309953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.310040 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.310404 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.310408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.310490 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.310581 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.310844 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":100,"thread_start_us":84,"threads_started":1}
I20260928 02:46:12.310987 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.302259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.27.109.190:38605)
0928 02:46:12.302488 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.302499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.302606 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:46:12.303141 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.303144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.303170 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.303265 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.303788 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.304599 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.305456 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.305776 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.305910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.307930 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.307936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.307937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.307943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.309413 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.309797 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.309799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.309800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.309883 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.310106 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.310107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.310107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.310323 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.310412 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.310610 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.310750 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
I20260928 02:46:12.311756 9006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.312065 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.311971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34112)
0928 02:46:12.312007 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.312046 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 02:46:12.312194 8954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.189@KRBTEST.COM" created.
I20260928 02:46:12.312628 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.312521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34124)
0928 02:46:12.312567 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.312596 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:46:12.312757 9006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:39517: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
I20260928 02:46:12.325193 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:39517
--webserver_interface=127.27.109.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:12.361977 9006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } attempt: 1
I20260928 02:46:12.362419 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.362288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34136)
0928 02:46:12.362348 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.362392 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:46:12.362782 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.362684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34144)
0928 02:46:12.362730 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.362757 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:46:12.362874 9006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:39517: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
W20260928 02:46:12.404088 9013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:12.404306 9013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:12.404345 9013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:12.405499 9013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:12.405565 9013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:12.405589 9013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:12.405616 9013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:12.405646 9013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:12.407222 9013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:39517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:12.407490 9013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:12.407701 9013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:12.412369 9018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:12.412356 9019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:12.412415 9013 server_base.cc:1034] running on GCE node
W20260928 02:46:12.412387 9021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:12.412801 9013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:12.413079 9013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:12.414220 9013 hybrid_clock.cc:648] HybridClock initialized: now 1790563572414184 us; error 56 us; skew 500 ppm
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:12.416615 9013 init.cc:377] Logged in from keytab as kudu/127.27.109.189@KRBTEST.COM (short username kudu)
I20260928 02:46:12.417033 9013 webserver.cc:460] Webserver started at http://127.27.109.189:34265/ using document root <none> and password file <none>
I20260928 02:46:12.417217 9013 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:12.417258 9013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:12.417342 9013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:12.418298 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data/instance:
uuid: "108d5a6f109d4b59ae02791169d722da"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "efb41e498822f6eac4662cac2dbac445"
server_key_iv: "cd2315c753fedc689879eb8ff0ba5eeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.418675 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal/instance:
uuid: "108d5a6f109d4b59ae02791169d722da"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "efb41e498822f6eac4662cac2dbac445"
server_key_iv: "cd2315c753fedc689879eb8ff0ba5eeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.419863 9013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:12.420503 9028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:12.420696 9013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:46:12.420773 9013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "108d5a6f109d4b59ae02791169d722da"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "efb41e498822f6eac4662cac2dbac445"
server_key_iv: "cd2315c753fedc689879eb8ff0ba5eeb"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.420821 9013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:12.431002 9006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } attempt: 2
I20260928 02:46:12.431388 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.431254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34148)
0928 02:46:12.431313 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.431353 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:46:12.431727 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.431647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34158)
0928 02:46:12.431687 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.431711 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:46:12.431843 9006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:39517: Network error: Client connection negotiation failed: client connection to 127.27.109.189:39517: connect: Connection refused (error 111)
I20260928 02:46:12.433707 9013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:12.435945 9013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:12.436101 9013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:12.439910 9013 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.189:39517
I20260928 02:46:12.439939 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.189:39517 every 8 connection(s)
I20260928 02:46:12.440265 9013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
I20260928 02:46:12.441038 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.442453 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:12.449573 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9013
I20260928 02:46:12.449707 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-1/wal/instance
I20260928 02:46:12.449903 28086 external_mini_cluster.cc:1428] Setting key c59e3463a208dcc0ee4c06860790ee6f
I20260928 02:46:12.450753 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.443197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.27.109.190:38605)
0928 02:46:12.443275 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.443277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.443447 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:46:12.443916 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.443917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.443930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.443982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.444560 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.445039 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.445995 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.446198 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.446274 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.448187 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.448188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.448188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.448196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.449373 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.449762 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.449764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.449764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.449825 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.450090 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.450091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.450092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.450198 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.450267 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.450581 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.450658 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:12.450934 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.443154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:38605 (local address 127.0.0.1:37688)
0928 02:46:12.443328 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.443370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.443811 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.444052 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.444055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.444277 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.444459 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.444463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.445133 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.445134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.445899 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.445901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.446279 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.446281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.446404 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.446839 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.446847 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.448094 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.449463 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.449469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.449471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.449704 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.449893 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.449894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.449894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.450030 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.450270 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.450278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.450427 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.450560 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.450706 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 02:46:12.451552 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.188@KRBTEST.COM; defaulting to no policy
I20260928 02:46:12.458698 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.451844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34168)
0928 02:46:12.451897 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.451906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.451946 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.452179 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.452180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.452325 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.452365 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.452367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.453201 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.453202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.453682 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.453683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.454125 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.454126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.454189 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.454604 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.454608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.455894 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.457779 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.457780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.457780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.457932 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.458120 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.458120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.458120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.458167 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.458427 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.458428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.458504 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.458572 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.458642 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:46:12.458803 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.451899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.27.109.189:39517)
0928 02:46:12.452053 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.452057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.452071 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:46:12.452087 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.452088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.452098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.452179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.452449 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.453093 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.453762 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.454041 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.454117 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.455982 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.455990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.455990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.455998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.457701 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.458007 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.458010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.458010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.458060 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.458203 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.458204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.458204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.458361 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.458434 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.458601 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.458697 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 02:46:12.459410 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.459721 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.459605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:45990)
0928 02:46:12.459657 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.459702 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.188@KRBTEST.COM" created.
W20260928 02:46:12.459864 9032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
I20260928 02:46:12.460306 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.460220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46006)
0928 02:46:12.460267 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.460291 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:46:12.460417 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:42081: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
I20260928 02:46:12.472949 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:42081
--webserver_interface=127.27.109.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:12.509675 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } attempt: 1
I20260928 02:46:12.510216 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.510069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46014)
0928 02:46:12.510138 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.510184 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 02:46:12.510555 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.510461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46026)
0928 02:46:12.510503 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.510533 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:46:12.510646 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:42081: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
I20260928 02:46:12.523015 9006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } attempt: 3
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:46:12.530812 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.523332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34178)
0928 02:46:12.523404 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.523416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.523504 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.523678 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.523679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.523856 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.523924 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.523926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.524909 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.524910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.525414 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.525415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.525883 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.525884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.525939 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.526421 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.526426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.527872 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.529787 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.529788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.529788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.529960 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.530167 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.530168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.530168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.530221 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.530545 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.530546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.530587 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.530643 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.530738 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:12.530879 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.523382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.27.109.189:39517)
0928 02:46:12.523442 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.523444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.523459 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:46:12.523576 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.523577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.523589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.523667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.524001 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.524822 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.525492 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.525811 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.525880 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.527964 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.527965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.527965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.527972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.529688 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.530033 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.530035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.530035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.530097 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.530296 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.530297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.530298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.530426 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.530512 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.530696 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.530790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:46:12.531551 9006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.531883 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.531785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46038)
0928 02:46:12.531830 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.531861 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:46:12.532238 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.532160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46048)
0928 02:46:12.532201 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.532224 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:46:12.532372 9006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:42081: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
W20260928 02:46:12.554665 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:12.554847 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:12.554868 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:12.555926 9088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:12.555960 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:12.555975 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:12.555989 9088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:12.556000 9088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:12.557418 9088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:38605,127.27.109.189:39517,127.27.109.188:42081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:42081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:12.557595 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:12.557749 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:12.562392 9094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:12.562424 9093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:12.562419 9096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:12.562536 9088 server_base.cc:1034] running on GCE node
I20260928 02:46:12.562813 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:12.563086 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:12.563524 9006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } attempt: 1
I20260928 02:46:12.563871 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.563756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46062)
0928 02:46:12.563809 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.563836 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:46:12.564136 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.564057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46078)
0928 02:46:12.564096 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.564117 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:46:12.564246 9088 hybrid_clock.cc:648] HybridClock initialized: now 1790563572564228 us; error 25 us; skew 500 ppm
W20260928 02:46:12.564256 9006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:42081: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:12.566680 9088 init.cc:377] Logged in from keytab as kudu/127.27.109.188@KRBTEST.COM (short username kudu)
I20260928 02:46:12.567092 9088 webserver.cc:460] Webserver started at http://127.27.109.188:34625/ using document root <none> and password file <none>
I20260928 02:46:12.567351 9088 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:12.567422 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:12.567564 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:12.568549 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data/instance:
uuid: "348bc3aa99304c999646f9f123d19062"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "6f803df7843acdcb00c3add95c0749e8"
server_key_iv: "bf6df6417cb512a6471daf324659c86c"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.568904 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal/instance:
uuid: "348bc3aa99304c999646f9f123d19062"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "6f803df7843acdcb00c3add95c0749e8"
server_key_iv: "bf6df6417cb512a6471daf324659c86c"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.570151 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:46:12.570945 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:12.571152 9088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:12.571242 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "348bc3aa99304c999646f9f123d19062"
format_stamp: "Formatted at 2026-09-28 02:46:12 on dist-test-slave-b5p5"
server_key: "6f803df7843acdcb00c3add95c0749e8"
server_key_iv: "bf6df6417cb512a6471daf324659c86c"
server_key_version: "encryptionkey@0"
I20260928 02:46:12.571329 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:12.578770 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } attempt: 2
I20260928 02:46:12.579123 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.579002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46094)
0928 02:46:12.579055 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.579098 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:12.579430 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.579358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46104)
0928 02:46:12.579385 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.579413 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 02:46:12.579514 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:42081: Network error: Client connection negotiation failed: client connection to 127.27.109.188:42081: connect: Connection refused (error 111)
I20260928 02:46:12.587541 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:12.589365 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:12.589514 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:12.593408 9088 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.188:42081
I20260928 02:46:12.593529 9155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.188:42081 every 8 connection(s)
I20260928 02:46:12.593802 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
I20260928 02:46:12.594859 9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:12.596675 9156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.597105 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9088
I20260928 02:46:12.597224 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563399140758-28086-0/minicluster-data/master-2/wal/instance
I20260928 02:46:12.597496 28086 external_mini_cluster.cc:1428] Setting key 45aa17ddae10e7e12ae987f3762d63c2
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:46:12.606269 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.597386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.27.109.190:38605)
0928 02:46:12.597453 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.597455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.597620 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:46:12.598238 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.598240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.598252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.598314 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.598918 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.599422 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.600298 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.600497 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.600571 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.602771 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.602772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.602772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.602778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.604708 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.605104 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.605106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.605106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.605189 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.605555 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.605556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.605557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.605686 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.605784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.606062 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.606151 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:12.606514 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.597363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:38605 (local address 127.0.0.1:37698)
0928 02:46:12.597507 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.597545 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.598133 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.598334 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.598337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.598658 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.598807 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.598811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.599500 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.599501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.600188 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.600190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.600587 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.600588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.600718 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.601214 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.601223 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.602694 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.604793 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.604799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.604800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.605016 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.605325 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.605326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.605327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.605437 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.605765 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.605772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.605909 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.606039 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.606193 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260928 02:46:12.606702 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.597881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.27.109.188:42081)
0928 02:46:12.598046 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.598051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.598083 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:46:12.598174 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.598189 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.598207 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.598300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.598965 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.599734 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.600465 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.600772 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.600857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.603462 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.603470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.603470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.603477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.605267 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.605599 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.605602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.605602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.605659 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.605870 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.605871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.605871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.606005 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.606153 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.606235 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.606324 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260928 02:46:12.607122 9156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:12.614380 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.607402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:39517 (local address 127.0.0.1:34182)
0928 02:46:12.607513 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.607524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.607577 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.607713 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.607714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.607875 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.607937 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.607939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.608682 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.608683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.609412 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.609414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.609739 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.609740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.609812 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.610273 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.610276 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.611681 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.613349 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.613350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.613351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.613528 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.613752 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.613753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.613754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.613827 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.614135 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.614136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.614170 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.614225 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.614287 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:46:12.614502 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.607468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.27.109.189:39517)
0928 02:46:12.607519 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.607521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.607562 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:46:12.607645 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.607647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.607656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.607709 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.608013 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.608602 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.609494 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.609669 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.609718 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.611769 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.611770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.611770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.611777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.613240 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.613608 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.613609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.613610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.613670 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.613898 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.613899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.613900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.614047 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.614125 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.614282 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.614368 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:46:12.614981 9156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:46:12.615181 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.607477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.27.109.190:38605)
0928 02:46:12.607524 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.607526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.607563 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:46:12.607642 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.607642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.607652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.607709 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.608300 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.608928 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.609850 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.610016 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.610064 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.612612 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.612613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.612614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.612620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.613979 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.614376 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.614377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.614377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.614432 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.614670 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.614671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.614672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.614805 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.614987 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.615025 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.615091 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:46:12.622071 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.615340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46122)
0928 02:46:12.615376 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.615386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.615424 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.615591 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.615592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.615702 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.615728 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.615729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.616483 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.616484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.616984 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.616986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.617411 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.617412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.617462 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.617897 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.617902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.619157 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.621003 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.621004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.621004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.621176 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.621401 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.621402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.621403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.621459 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.621720 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.621721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.621759 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.621860 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.621955 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 02:46:12.622138 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.615374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.27.109.188:42081)
0928 02:46:12.615487 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.615489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.615495 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:46:12.615513 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.615514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.615524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.615570 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.615815 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.616390 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.617069 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.617320 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.617411 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.619262 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.619263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.619264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.619271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.620920 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.621266 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.621267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.621267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.621331 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.621546 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.621547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.621547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.621651 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.621727 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.621884 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.622010 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":100}
I20260928 02:46:12.623426 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.616073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.27.109.189:39517)
0928 02:46:12.616132 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.616133 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.616140 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:12.616157 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.616158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.616166 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.616220 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.616660 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.617096 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.617833 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.618177 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.618255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.620653 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.620654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.620654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.620660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.622019 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.622602 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.622603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.622604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.622676 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.622881 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.622882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.622883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.622988 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.623143 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.623186 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.623252 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:12.623647 9156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: Bootstrap starting.
I20260928 02:46:12.624804 9156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.625168 9156 log.cc:824] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:12.625283 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:12.625941 9156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: No bootstrap required, opened a new log
I20260928 02:46:12.628230 9156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.628388 9156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.628437 9156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348bc3aa99304c999646f9f123d19062, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.628561 9156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.628875 9156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:12.629809 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.629935 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:46:12.630766 9180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:12.634682 9180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:12.634765 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.626672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.27.109.189:39517)
0928 02:46:12.626719 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.626720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.627093 (+ 373us) server_negotiation.cc:407] Connection header received
0928 02:46:12.627152 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.627153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.627161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.627224 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.627712 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.628370 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.629221 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.629500 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.629569 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.631401 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.631402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.631403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.631409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.633148 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.633535 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.633536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.633537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.633588 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.634064 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.634065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.634065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.634180 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.634263 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.634591 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.634666 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:46:12.632166 9088 master_runner.cc:186] Error getting master registration for 127.27.109.190:38605: OK, instance_id { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" instance_seqno: 1790563572297624 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:46:12.635524 9088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:46:12.638025 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.626760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.27.109.188:42081)
0928 02:46:12.630307 (+ 3547us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.630309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.630512 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:46:12.630623 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.630624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.630633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.630663 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.631231 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.631871 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.632923 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.633283 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.633329 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.634808 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.634809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.634809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.634816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.636544 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.636993 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.636995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.636995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.637050 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.637295 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.637296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.637296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.637478 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.637564 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.637824 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.637930 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3530}
I20260928 02:46:12.638155 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.626747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.27.109.190:38605)
0928 02:46:12.626799 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.626801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.627005 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:46:12.627052 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.627053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.627062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.627125 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.627915 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.628576 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.629723 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.630134 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.630458 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.631555 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.631556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.631556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.631563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.636863 (+ 5300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.637260 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.637261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.637262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.637346 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.637570 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.637571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.637571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.637704 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.637801 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.637952 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.638025 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:46:12.639405 9006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } attempt: 2
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:46:12.646350 9008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.639635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46138)
0928 02:46:12.639686 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.639696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.639733 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.639886 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.639886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.639999 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.640028 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.640029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.640896 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.640897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.641420 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.641421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.641866 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.641867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.641947 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.642369 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.642373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.643800 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.645414 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.645415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.645415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.645574 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.645793 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.645793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.645794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.645844 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.646107 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.646108 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.646148 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.646216 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.646293 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:12.646417 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.639697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.27.109.188:42081)
0928 02:46:12.639728 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.639730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.639737 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:12.639812 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.639813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.639821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.639883 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.640097 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.640812 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.641510 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.641788 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.641867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.643879 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.643880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.643881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.643887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.645334 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.645646 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.645647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.645647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.645701 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.645904 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.645905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.645905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.646020 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.646086 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.646266 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.646335 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:46:12.647934 9006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: Bootstrap starting.
I20260928 02:46:12.648618 9006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.648919 9006 log.cc:824] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:12.649735 9006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: No bootstrap required, opened a new log
I20260928 02:46:12.651198 9006 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.651327 9006 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.651373 9006 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca344a8119bf4147b7ffd16868691ab6, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.651518 9006 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.651815 9006 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:12.651803 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.651902 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:46:12.653534 9194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:46:12.653630 9194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:12.654248 8938 master_runner.cc:186] Error getting master registration for 127.27.109.189:39517: OK, instance_id { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" instance_seqno: 1790563572438800 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:46:12.656682 8938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:46:12.659740 9183 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:12.659868 9183 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.660140 9183 leader_election.cc:288] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 108d5a6f109d4b59ae02791169d722da (127.27.109.189:39517), 348bc3aa99304c999646f9f123d19062 (127.27.109.188:42081)
I20260928 02:46:12.660411 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca344a8119bf4147b7ffd16868691ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348bc3aa99304c999646f9f123d19062" is_pre_election: true
I20260928 02:46:12.660567 9130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca344a8119bf4147b7ffd16868691ab6 in term 0.
I20260928 02:46:12.660782 8957 leader_election.cc:302] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348bc3aa99304c999646f9f123d19062, ca344a8119bf4147b7ffd16868691ab6; no voters:
I20260928 02:46:12.660915 9183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:12.660969 9183 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:12.661000 9183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.661769 9183 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.661896 9183 leader_election.cc:288] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [CANDIDATE]: Term 1 election: Requested vote from peers 108d5a6f109d4b59ae02791169d722da (127.27.109.189:39517), 348bc3aa99304c999646f9f123d19062 (127.27.109.188:42081)
I20260928 02:46:12.662158 9130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca344a8119bf4147b7ffd16868691ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348bc3aa99304c999646f9f123d19062"
I20260928 02:46:12.662232 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.663147 9130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca344a8119bf4147b7ffd16868691ab6 in term 1.
I20260928 02:46:12.663328 8957 leader_election.cc:302] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348bc3aa99304c999646f9f123d19062, ca344a8119bf4147b7ffd16868691ab6; no voters:
I20260928 02:46:12.663405 9183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:12.663549 9183 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [term 1 LEADER]: Becoming Leader. State: Replica: ca344a8119bf4147b7ffd16868691ab6, State: Running, Role: LEADER
I20260928 02:46:12.663651 9183 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.664001 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca344a8119bf4147b7ffd16868691ab6. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.664089 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:12.664309 9198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:12.664528 9198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:12.666128 9130 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [term 1 FOLLOWER]: Refusing update from remote peer ca344a8119bf4147b7ffd16868691ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.666519 9183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:12.667775 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca344a8119bf4147b7ffd16868691ab6. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.667861 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:12.668929 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.669014 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:12.669072 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.669143 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:12.669116 9199 mvcc.cc:205] Tried to move back new op lower bound from 7334148393638162432 to 7334148393630740480. Current Snapshot: MvccSnapshot[applied={T|T < 7334148393638162432}]
I20260928 02:46:12.669495 9200 mvcc.cc:205] Tried to move back new op lower bound from 7334148393638162432 to 7334148393630740480. Current Snapshot: MvccSnapshot[applied={T|T < 7334148393638162432}]
I20260928 02:46:12.670033 9198 catalog_manager.cc:1347] Generated new cluster ID: 2cbada2f9b80459ab070fc3cebb57678
I20260928 02:46:12.670081 9198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:12.674610 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } attempt: 3
Sep 28 02:46:12 dist-test-slave-b5p5 krb5kdc[8925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563572, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:46:12.682813 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.674996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:42081 (local address 127.0.0.1:46148)
0928 02:46:12.675079 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:12.675091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:12.675138 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:12.675381 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:12.675382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:12.675578 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:12.675629 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.675631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.676527 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.676528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.677302 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:12.677304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.677820 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.677821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:12.677889 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.678364 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:12.678370 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:12.679904 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:12.681712 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.681713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.681713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.681919 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.682144 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:12.682145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:12.682145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:12.682212 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:12.682521 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:12.682521 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:12.682579 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:46:12.682635 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:46:12.682721 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 02:46:12.683260 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:12.675046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.27.109.188:42081)
0928 02:46:12.675091 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:12.675094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:12.675124 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:46:12.675191 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:12.675192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:12.675202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:12.675270 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:12.675685 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.676451 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.677404 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:12.677700 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:12.677751 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:12.680003 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:12.680005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:12.680005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:12.680011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:12.681608 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.682001 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.682002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.682002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.682066 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:12.682289 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:12.682289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:12.682290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:12.682414 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:12.682481 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:12.683103 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 02:46:12.683176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:46:12.685007 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: Bootstrap starting.
I20260928 02:46:12.685720 9081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:12.686040 9081 log.cc:824] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:12.687410 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: No bootstrap required, opened a new log
I20260928 02:46:12.689479 9081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.689610 9081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:12.689646 9081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108d5a6f109d4b59ae02791169d722da, State: Initialized, Role: FOLLOWER
I20260928 02:46:12.689788 9081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } }
I20260928 02:46:12.690091 9081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:12.690151 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca344a8119bf4147b7ffd16868691ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108d5a6f109d4b59ae02791169d722da"
I20260928 02:46:12.690241 9054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:12.691191 9054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca344a8119bf4147b7ffd16868691ab6 in term 1.
I20260928 02:46:12.691412 9053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 1 FOLLOWER]: Refusing update from remote peer ca344a8119bf4147b7ffd16868691ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:12.691555 9055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca344a8119bf4147b7ffd16868691ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108d5a6f109d4b59ae02791169d722da" is_pre_election: true
I20260928 02:46:12.692049 9055 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ca344a8119bf4147b7ffd16868691ab6 in term 1. Re-sending same reply.
I20260928 02:46:12.692579 9196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:12.694146 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca344a8119bf4147b7ffd16868691ab6. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.694293 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:12.694397 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.694538 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:12.695487 9218 mvcc.cc:205] Tried to move back new op lower bound from 7334148393638162432 to 7334148393630740480. Current Snapshot: MvccSnapshot[applied={T|T < 7334148393638162432}]
I20260928 02:46:12.699345 9216 catalog_manager.cc:1259] Loaded cluster ID: 2cbada2f9b80459ab070fc3cebb57678
I20260928 02:46:12.699424 9216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: loading cluster ID for follower catalog manager: success
W20260928 02:46:12.699535 9216 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 02:46:12.699601 9216 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:46:12.690091 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.699792 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:46:12.707836 9196 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca344a8119bf4147b7ffd16868691ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca344a8119bf4147b7ffd16868691ab6" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 38605 } } peers { permanent_uuid: "108d5a6f109d4b59ae02791169d722da" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 39517 } } peers { permanent_uuid: "348bc3aa99304c999646f9f123d19062" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 42081 } } }
I20260928 02:46:12.707942 9196 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:12.709211 9198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:12.709924 9198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:12.722441 9198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca344a8119bf4147b7ffd16868691ab6: Generated new TSK 0
I20260928 02:46:12.722671 9198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:13.635192 9180 catalog_manager.cc:1259] Loaded cluster ID: 2cbada2f9b80459ab070fc3cebb57678
I20260928 02:46:13.635265 9180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: loading cluster ID for follower catalog manager: success
I20260928 02:46:13.636009 9180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: acquiring CA information for follower catalog manager: success
I20260928 02:46:13.636245 9180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 348bc3aa99304c999646f9f123d19062: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:46:13.700690 9216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: acquiring CA information for follower catalog manager: success
I20260928 02:46:13.701047 9216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 108d5a6f109d4b59ae02791169d722da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:46:13.782241 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 8938
I20260928 02:46:13.787335 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9013
I20260928 02:46:13.792061 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9088
2026-09-28T02:46:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2752 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:13 dist-test-slave-b5p5 krb5kdc[9230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:13 dist-test-slave-b5p5 krb5kdc[9230](info): set up 2 sockets
Sep 28 02:46:13 dist-test-slave-b5p5 krb5kdc[9230](info): commencing operation
krb5kdc: starting...
W20260928 02:46:14.814634 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:46:14 dist-test-slave-b5p5 krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:46:14.925091 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37855
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:15.003779 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:15.003970 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:15.003993 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:15.005080 9243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:15.005120 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:15.005136 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:15.005147 9243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:15.005159 9243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:15.006628 9243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:15.006847 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:15.007066 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:15.011996 9251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:15.012028 9248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:15.012072 9249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:15.012074 9243 server_base.cc:1034] running on GCE node
I20260928 02:46:15.012359 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:15.012571 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:15.013743 9243 hybrid_clock.cc:648] HybridClock initialized: now 1790563575013725 us; error 40 us; skew 500 ppm
Sep 28 02:46:15 dist-test-slave-b5p5 krb5kdc[9230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563575, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:15.016207 9243 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:15.016623 9243 webserver.cc:460] Webserver started at http://127.27.109.190:38419/ using document root <none> and password file <none>
I20260928 02:46:15.016798 9243 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:15.016837 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:15.016919 9243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:15.017904 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "9fe1dc6536d44186a173cc6975dde767"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-b5p5"
server_key: "3ce4c40de624d7654f0fcff7321f84bf"
server_key_iv: "de73f1d6b3be01ff867e0fc752cb263c"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.018206 9243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "9fe1dc6536d44186a173cc6975dde767"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-b5p5"
server_key: "3ce4c40de624d7654f0fcff7321f84bf"
server_key_iv: "de73f1d6b3be01ff867e0fc752cb263c"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.019493 9243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:46:15.020154 9258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:15.020344 9243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:15.020412 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "9fe1dc6536d44186a173cc6975dde767"
format_stamp: "Formatted at 2026-09-28 02:46:15 on dist-test-slave-b5p5"
server_key: "3ce4c40de624d7654f0fcff7321f84bf"
server_key_iv: "de73f1d6b3be01ff867e0fc752cb263c"
server_key_version: "encryptionkey@0"
I20260928 02:46:15.020462 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:15.038957 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:15.041115 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:15.041219 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:15.044981 9243 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37855
I20260928 02:46:15.045030 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37855 every 8 connection(s)
I20260928 02:46:15.045367 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:15.046149 9311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:15.048280 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767: Bootstrap starting.
I20260928 02:46:15.048818 9311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:15.049041 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9243
I20260928 02:46:15.049154 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:46:15.049166 9311 log.cc:824] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:15.049362 28086 external_mini_cluster.cc:1428] Setting key 16ceee27cc0efd4f6525e5dd1835ae95
I20260928 02:46:15.049937 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767: No bootstrap required, opened a new log
I20260928 02:46:15.052166 9311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } }
I20260928 02:46:15.052330 9311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:15.052377 9311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe1dc6536d44186a173cc6975dde767, State: Initialized, Role: FOLLOWER
I20260928 02:46:15.052497 9311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } }
I20260928 02:46:15.052591 9311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:15.052634 9311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:15.052680 9311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:15.053180 9311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } }
I20260928 02:46:15.053305 9311 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe1dc6536d44186a173cc6975dde767; no voters:
I20260928 02:46:15.053480 9311 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:15.053522 9316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:15.053656 9316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [term 1 LEADER]: Becoming Leader. State: Replica: 9fe1dc6536d44186a173cc6975dde767, State: Running, Role: LEADER
I20260928 02:46:15.053735 9311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:15.053747 9316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } }
I20260928 02:46:15.054107 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe1dc6536d44186a173cc6975dde767. Latest consensus state: current_term: 1 leader_uuid: "9fe1dc6536d44186a173cc6975dde767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } } }
I20260928 02:46:15.054180 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:15.054332 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fe1dc6536d44186a173cc6975dde767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe1dc6536d44186a173cc6975dde767" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37855 } } }
I20260928 02:46:15.054409 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:15.054594 9320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:15.055094 9320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:46:15 dist-test-slave-b5p5 krb5kdc[9230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:15.056770 9320 catalog_manager.cc:1347] Generated new cluster ID: fd58554a159d4557af6e188f6b2b983d
I20260928 02:46:15.057045 9320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:15.059924 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:15.049817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.27.109.190:37855)
0928 02:46:15.050004 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:15.050014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:15.050044 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:46:15.050431 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:15.050437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:15.050559 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:15.050688 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:15.051313 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:15.052186 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:15.053086 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:15.053642 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:15.054037 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:15.056554 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:15.056558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:15.056558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:15.056571 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:15.058370 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:15.058767 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:15.058770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:15.058770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:15.058868 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:15.059066 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:15.059067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:15.059067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:15.059351 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:15.059522 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:15.059598 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:46:15.059724 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260928 02:46:15.060803 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:46:15.072585 9320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:15.073247 9320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:15.081743 9320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fe1dc6536d44186a173cc6975dde767: Generated new TSK 0
I20260928 02:46:15.081904 9320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:46:15.114503 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9243
2026-09-28T02:46:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1320 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:15 dist-test-slave-b5p5 krb5kdc[9338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:15 dist-test-slave-b5p5 krb5kdc[9338](info): set up 2 sockets
Sep 28 02:46:15 dist-test-slave-b5p5 krb5kdc[9338](info): commencing operation
krb5kdc: starting...
W20260928 02:46:16.139047 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:46:16.247217 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.324550 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.324733 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.324753 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.325856 9351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:16.325897 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.325911 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.325924 9351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:16.325959 9351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:16.327409 9351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:16.327642 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.327850 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.332511 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.332499 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.332501 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.332727 9351 server_base.cc:1034] running on GCE node
I20260928 02:46:16.332938 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.333148 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.334280 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790563576334249 us; error 45 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.336709 9351 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:16.337145 9351 webserver.cc:460] Webserver started at http://127.27.109.190:42565/ using document root <none> and password file <none>
I20260928 02:46:16.337359 9351 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.337404 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.337541 9351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.338580 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "526c5fd853ea4dbeacd46acb147fd2a3"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "b98bbc1e31af99a3247fea798f763966"
server_key_iv: "85187346d86f6a0569fc2ffbb5393b7f"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.338920 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "526c5fd853ea4dbeacd46acb147fd2a3"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "b98bbc1e31af99a3247fea798f763966"
server_key_iv: "85187346d86f6a0569fc2ffbb5393b7f"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.340140 9351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:16.340813 9366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.340983 9351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:46:16.341050 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "526c5fd853ea4dbeacd46acb147fd2a3"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "b98bbc1e31af99a3247fea798f763966"
server_key_iv: "85187346d86f6a0569fc2ffbb5393b7f"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.341136 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.352288 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.354519 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.354686 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.358604 9351 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37383
I20260928 02:46:16.358632 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37383 every 8 connection(s)
I20260928 02:46:16.358946 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:16.359812 9419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:16.360992 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9351
I20260928 02:46:16.361125 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:46:16.361336 28086 external_mini_cluster.cc:1428] Setting key 93a196341b85b3890e55c053a55c134c
I20260928 02:46:16.362334 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap starting.
I20260928 02:46:16.363047 9419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:16.363387 9419 log.cc:824] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:16.364046 9419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: No bootstrap required, opened a new log
I20260928 02:46:16.365886 9419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:16.366036 9419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:16.366083 9419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Initialized, Role: FOLLOWER
I20260928 02:46:16.366209 9419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:16.366312 9419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:16.366344 9419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:16.366386 9419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:16.367122 9419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:16.367237 9419 leader_election.cc:302] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526c5fd853ea4dbeacd46acb147fd2a3; no voters:
I20260928 02:46:16.367408 9419 leader_election.cc:288] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:16.367574 9425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:16.367654 9419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:16.367707 9425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 LEADER]: Becoming Leader. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Running, Role: LEADER
I20260928 02:46:16.367794 9425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:16.368140 9424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:16.368233 9424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:16.368124 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 526c5fd853ea4dbeacd46acb147fd2a3. Latest consensus state: current_term: 1 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:16.368413 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:16.368525 9433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:16.369254 9433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:16.371153 9433 catalog_manager.cc:1347] Generated new cluster ID: 2955d2394050405da50c61f3ace2ea19
I20260928 02:46:16.371204 9433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:16.372022 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.361814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.27.109.190:37383)
0928 02:46:16.361999 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.362007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.362041 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:16.362498 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.362506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.362574 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.362681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.363509 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.364439 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.365193 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.365582 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.365696 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.368582 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.368586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.368586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.368597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.370507 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.370843 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.370846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.370846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.370945 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.371175 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.371176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.371177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.371424 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.371570 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.371676 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.371818 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260928 02:46:16.403124 9433 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:16.403606 9433 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:16.408598 9433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Generated new TSK 0
I20260928 02:46:16.408769 9433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:46:16.441618 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.519822 9445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.520025 9445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.520046 9445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.521082 9445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.521145 9445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.521193 9445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:16.522605 9445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:16.522822 9445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.523051 9445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.527724 9453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.527706 9451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.527699 9450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.527956 9445 server_base.cc:1034] running on GCE node
I20260928 02:46:16.528172 9445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.528401 9445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.529577 9445 hybrid_clock.cc:648] HybridClock initialized: now 1790563576529439 us; error 148 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.531998 9445 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:16.532392 9445 webserver.cc:460] Webserver started at http://127.27.109.129:44433/ using document root <none> and password file <none>
I20260928 02:46:16.532634 9445 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.532716 9445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.532840 9445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.533875 9445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "6146a8c98cfc4846995b85524ab839b9"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "ac1eda41151054561e7c419a6f981734"
server_key_iv: "bf852c2585bc90eb97061d7f49676b45"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.534245 9445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "6146a8c98cfc4846995b85524ab839b9"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "ac1eda41151054561e7c419a6f981734"
server_key_iv: "bf852c2585bc90eb97061d7f49676b45"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.535573 9445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:46:16.536253 9460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.536422 9445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.536512 9445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6146a8c98cfc4846995b85524ab839b9"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "ac1eda41151054561e7c419a6f981734"
server_key_iv: "bf852c2585bc90eb97061d7f49676b45"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.536593 9445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.556785 9445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.559213 9445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.559378 9445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.559620 9445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.559922 9445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.559978 9445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.560012 9445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.560041 9445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.565760 9445 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:39387
I20260928 02:46:16.565825 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:39387 every 8 connection(s)
I20260928 02:46:16.566147 9445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:16.574442 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.566829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:34629 (local address 127.27.109.190:37383)
0928 02:46:16.566876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.566878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.567076 (+ 198us) server_negotiation.cc:407] Connection header received
0928 02:46:16.567552 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.567553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.567571 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.567658 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.568575 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.569113 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.570022 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.570214 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.570262 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.572246 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.572247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.572247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.572254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.573383 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.573719 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.573720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.573720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.573766 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.573974 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.573974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.573974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.574065 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.574113 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.574320 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.574381 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:46:16.574667 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.566826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:34629)
0928 02:46:16.566967 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.567000 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.567459 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.567710 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.567714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.568241 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.568498 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.568503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.569188 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.569189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.569957 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.569958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.570281 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.570282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.570372 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.570900 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.570916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.572170 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.573452 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.573458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.573459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.573660 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.573821 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.573822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.573822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.573918 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.574117 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.574121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.574221 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.574297 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.574403 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260928 02:46:16.575250 9574 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:16.575366 9574 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:16.575531 9574 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:16.575898 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9445
I20260928 02:46:16.576014 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:16.576112 9383 ts_manager.cc:194] Registered new tserver with Master: 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387)
I20260928 02:46:16.576216 28086 external_mini_cluster.cc:1428] Setting key 8634f06b3f3a7e7c34566bb045b23d1e
I20260928 02:46:16.576992 9383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:34629
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:46:16.598886 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.677371 9579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.677559 9579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.677582 9579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.678653 9579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.678686 9579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.678711 9579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:16.680012 9579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:16.680173 9579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.680328 9579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.685038 9587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.685016 9584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.685065 9579 server_base.cc:1034] running on GCE node
W20260928 02:46:16.685032 9585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.685431 9579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.685621 9579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.686750 9579 hybrid_clock.cc:648] HybridClock initialized: now 1790563576686738 us; error 26 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.689014 9579 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:16.689446 9579 webserver.cc:460] Webserver started at http://127.27.109.130:39951/ using document root <none> and password file <none>
I20260928 02:46:16.689661 9579 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.689703 9579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.689867 9579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.690881 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "0bcb6f9f5704469e955955d03d52899c"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "834bf54a85c62bf63e35330ab3264aea"
server_key_iv: "f205f3bede9a8158177e3cbe1c0a2eb5"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.691222 9579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "0bcb6f9f5704469e955955d03d52899c"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "834bf54a85c62bf63e35330ab3264aea"
server_key_iv: "f205f3bede9a8158177e3cbe1c0a2eb5"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.692683 9579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:16.693639 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.693781 9579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:16.693866 9579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "0bcb6f9f5704469e955955d03d52899c"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "834bf54a85c62bf63e35330ab3264aea"
server_key_iv: "f205f3bede9a8158177e3cbe1c0a2eb5"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.693943 9579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.704100 9579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.705808 9579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.705953 9579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.706153 9579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.706441 9579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.706570 9579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.706624 9579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.706650 9579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.712087 9579 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:46687
I20260928 02:46:16.712158 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:46687 every 8 connection(s)
I20260928 02:46:16.712440 9579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:16.712796 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9579
I20260928 02:46:16.712911 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:16.713138 28086 external_mini_cluster.cc:1428] Setting key a961df60afec01dc141f1920990c60c0
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:16.721891 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.713221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34973 (local address 127.27.109.190:37383)
0928 02:46:16.713288 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.713290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.713652 (+ 362us) server_negotiation.cc:407] Connection header received
0928 02:46:16.714201 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.714203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.714215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.714264 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.715194 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.715690 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.716666 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.716943 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.717029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.719084 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.719085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.719086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.719093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.720473 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.720907 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.720908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.720909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.720966 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.721209 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.721211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.721211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.721325 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.721400 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.721724 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.721804 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:46:16.722069 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.713149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:34973)
0928 02:46:16.713536 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.713577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.714096 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.714325 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.714328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.714934 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.715101 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.715105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.715796 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.715797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.716578 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.716580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.717036 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.717037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.717163 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.717646 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.717654 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.718985 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.720554 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.720561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.720563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.720826 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.721034 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.721034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.721035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.721141 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.721391 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.721428 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.721577 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.721701 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.721844 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":81,"threads_started":1}
I20260928 02:46:16.722554 9708 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:16.722662 9708 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
I20260928 02:46:16.722828 9708 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:16.723217 9383 ts_manager.cc:194] Registered new tserver with Master: 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
I20260928 02:46:16.723742 9383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:34973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:46:16.735122 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:16.814843 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:16.815059 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:16.815094 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:16.816205 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:16.816268 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:16.816313 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:16.817698 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:16.817950 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:16.818174 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:16.822959 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.822991 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:16.823032 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:16.823201 9713 server_base.cc:1034] running on GCE node
I20260928 02:46:16.823397 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:16.823633 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:16.824791 9713 hybrid_clock.cc:648] HybridClock initialized: now 1790563576824778 us; error 24 us; skew 500 ppm
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:16.827049 9713 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:16.827464 9713 webserver.cc:460] Webserver started at http://127.27.109.131:40281/ using document root <none> and password file <none>
I20260928 02:46:16.827694 9713 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:16.827759 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:16.827903 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:16.828873 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "cf4b8eab51db4474a6631d696d145966"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "01ef11aaa8314676c7b86156943524a5"
server_key_iv: "95214030b5f0b3f087dbb276beff0892"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.829212 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "cf4b8eab51db4474a6631d696d145966"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "01ef11aaa8314676c7b86156943524a5"
server_key_iv: "95214030b5f0b3f087dbb276beff0892"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.830421 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:16.831137 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.831305 9713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:16.831400 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "cf4b8eab51db4474a6631d696d145966"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "01ef11aaa8314676c7b86156943524a5"
server_key_iv: "95214030b5f0b3f087dbb276beff0892"
server_key_version: "encryptionkey@0"
I20260928 02:46:16.831488 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:16.855444 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:16.857157 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:16.857316 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:16.857532 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:16.857818 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:16.857869 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.857903 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:16.857929 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:16.863457 9713 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:42595
I20260928 02:46:16.863545 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:42595 every 8 connection(s)
I20260928 02:46:16.863829 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:16.869268 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9713
I20260928 02:46:16.869377 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:16.869571 28086 external_mini_cluster.cc:1428] Setting key 2bc53b80821b6c5ced924b7cbe1f0e8f
Sep 28 02:46:16 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563576, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:16.873006 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.864496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41471 (local address 127.27.109.190:37383)
0928 02:46:16.864537 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.864539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.864735 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:46:16.865231 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.865231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.865240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.865295 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.866153 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.866696 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.867680 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.867858 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.867910 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.870373 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.870374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.870375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.870382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.871863 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.872319 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.872320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.872320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.872368 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.872540 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.872541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.872541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.872635 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.872684 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.872885 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.872937 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:46:16.873243 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.864469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:41471)
0928 02:46:16.864629 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.864669 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.865145 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.865369 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.865372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.865890 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:16.866081 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.866084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.866768 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.866769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.867613 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.867615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.867926 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.867927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.868048 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.868550 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:16.868566 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:16.870264 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:16.871955 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.871962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.871964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.872256 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.872407 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:16.872408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:16.872408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:16.872493 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:16.872691 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:16.872695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:16.872798 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.872876 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.872994 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260928 02:46:16.873764 9842 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:16.873867 9842 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:16.874040 9842 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:16.874375 9383 ts_manager.cc:194] Registered new tserver with Master: cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595)
I20260928 02:46:16.874773 9383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:41471
I20260928 02:46:16.881157 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:16.887745 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.882407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.27.109.190:37383)
0928 02:46:16.882502 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.882504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.882598 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:46:16.882647 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.882647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.882655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.882702 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:16.883202 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.883666 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.884325 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.884496 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.884542 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.885545 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:16.885546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:16.885546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:16.885551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:16.886908 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.887179 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.887179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.887180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.887225 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:16.887372 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:16.887372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:16.887372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:16.887489 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:16.887558 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.887640 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.887691 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:16.889587 9383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:16.891921 9383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:16.899564 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:46687 (local address 127.0.0.1:56504)
0928 02:46:16.895000 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.895007 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.895035 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.895509 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.895512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.895514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:16.895579 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.895580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.897064 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.897065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.898231 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.898234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.899278 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.899279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.899355 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.899371 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.899417 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.899470 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":55,"threads_started":1}
I20260928 02:46:16.900321 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.27.109.130:46687)
0928 02:46:16.895251 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.895257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.895267 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:46:16.895311 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.895314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.895329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.895403 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:16.895677 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.896945 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.898290 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.899187 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.899969 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.900033 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.900058 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.900116 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":77,"threads_started":1}
I20260928 02:46:16.900996 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.27.109.131:42595)
0928 02:46:16.895719 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.895724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.895733 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:16.895769 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.895771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.895784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.896001 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:16.896061 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.898384 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.899988 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.900658 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.900737 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.900785 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.900888 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.900923 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":90,"threads_started":1}
I20260928 02:46:16.901006 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:42595 (local address 127.0.0.1:45234)
0928 02:46:16.894995 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.895006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.895061 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.895875 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.895876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.895878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:16.895928 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.895930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.898500 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.898501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.899699 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.899703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.900748 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.900749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.900811 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.900814 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.900876 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.900913 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260928 02:46:16.901193 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.27.109.129:39387)
0928 02:46:16.896262 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.896266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.896276 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:46:16.896311 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.896313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.896327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.896430 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:16.896563 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.897579 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.898986 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.900696 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.900786 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.900838 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.901047 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.901091 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":89,"threads_started":1}
I20260928 02:46:16.901317 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.894304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.0.0.1:41988)
0928 02:46:16.894783 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.894825 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.894889 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.896442 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.896443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.896445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:16.896497 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.896498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.897673 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.897674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.898683 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.898686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.900869 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.900871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.900922 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.900925 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.900971 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.901002 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":53,"threads_started":1}
I20260928 02:46:16.901201 9642 tablet_service.cc:1467] Processing CreateTablet for tablet a804093c9519465fa8db265885ef97ef (DEFAULT_TABLE table=test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:16.901628 9642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a804093c9519465fa8db265885ef97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:16.902046 9508 tablet_service.cc:1467] Processing CreateTablet for tablet a804093c9519465fa8db265885ef97ef (DEFAULT_TABLE table=test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:16.902364 9508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a804093c9519465fa8db265885ef97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:16.903824 9776 tablet_service.cc:1467] Processing CreateTablet for tablet a804093c9519465fa8db265885ef97ef (DEFAULT_TABLE table=test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:16.904140 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a804093c9519465fa8db265885ef97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:16.904461 9856 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Bootstrap starting.
I20260928 02:46:16.905130 9856 tablet_bootstrap.cc:654] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:16.905740 9856 log.cc:824] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:16.906749 9856 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: No bootstrap required, opened a new log
I20260928 02:46:16.906810 9856 ts_tablet_manager.cc:1397] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:16.907061 9857 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Bootstrap starting.
I20260928 02:46:16.907826 9857 tablet_bootstrap.cc:654] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:16.908384 9857 log.cc:824] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:16.909181 9857 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: No bootstrap required, opened a new log
I20260928 02:46:16.909245 9857 ts_tablet_manager.cc:1397] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:16.909168 9856 raft_consensus.cc:348] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.909334 9856 raft_consensus.cc:374] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:16.909375 9856 raft_consensus.cc:729] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb6f9f5704469e955955d03d52899c, State: Initialized, Role: FOLLOWER
I20260928 02:46:16.909487 9856 consensus_queue.cc:260] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.909744 9856 ts_tablet_manager.cc:1428] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:16.909863 9708 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
I20260928 02:46:16.910183 9859 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Bootstrap starting.
I20260928 02:46:16.910866 9859 tablet_bootstrap.cc:654] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:16.911420 9859 log.cc:824] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:16.911413 9857 raft_consensus.cc:348] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.911576 9857 raft_consensus.cc:374] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:16.911630 9857 raft_consensus.cc:729] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6146a8c98cfc4846995b85524ab839b9, State: Initialized, Role: FOLLOWER
I20260928 02:46:16.911744 9857 consensus_queue.cc:260] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.912020 9857 ts_tablet_manager.cc:1428] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:16.912019 9574 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
I20260928 02:46:16.912278 9859 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: No bootstrap required, opened a new log
I20260928 02:46:16.912345 9859 ts_tablet_manager.cc:1397] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:46:16.914342 9859 raft_consensus.cc:348] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.914521 9859 raft_consensus.cc:374] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:16.914575 9859 raft_consensus.cc:729] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4b8eab51db4474a6631d696d145966, State: Initialized, Role: FOLLOWER
I20260928 02:46:16.914697 9859 consensus_queue.cc:260] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.914986 9859 ts_tablet_manager.cc:1428] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:46:16.916349 9842 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
I20260928 02:46:16.918581 9861 raft_consensus.cc:482] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:16.918663 9861 raft_consensus.cc:504] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.919003 9861 leader_election.cc:288] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387)
I20260928 02:46:16.922873 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.919122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:42595 (local address 127.27.109.130:40491)
0928 02:46:16.919211 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.919222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.919263 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.919402 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.919403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.919404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:16.919465 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.919467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.920340 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.920341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.921660 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.921663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.922651 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.922651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.922708 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.922711 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.922776 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.922811 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:46:16.922870 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.919209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:40491 (local address 127.27.109.131:42595)
0928 02:46:16.919267 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.919269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.919306 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:46:16.919329 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.919330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.919339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.919393 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:16.919545 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.920256 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.921746 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.922248 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.922312 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.922318 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.922781 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.922815 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:46:16.922909 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.919302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:34937 (local address 127.27.109.129:39387)
0928 02:46:16.919338 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.919339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.919373 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:16.919460 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.919461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.919470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.919522 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:16.919705 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.920367 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.921967 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.922556 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.922618 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.922624 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.922813 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.922851 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:46:16.923110 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.919122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.27.109.130:34937)
0928 02:46:16.919319 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.919330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:16.919384 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:16.919538 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:16.919539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:16.919541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:16.919606 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.919607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.920566 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.920567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.921873 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:16.921877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.922636 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.922638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:16.922713 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.922717 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:16.923006 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 02:46:16.923044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":10,"thread_start_us":65,"threads_started":1}
I20260928 02:46:16.923153 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a804093c9519465fa8db265885ef97ef" candidate_uuid: "0bcb6f9f5704469e955955d03d52899c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4b8eab51db4474a6631d696d145966" is_pre_election: true
I20260928 02:46:16.923264 9796 raft_consensus.cc:2454] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcb6f9f5704469e955955d03d52899c in term 0.
I20260928 02:46:16.923377 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a804093c9519465fa8db265885ef97ef" candidate_uuid: "0bcb6f9f5704469e955955d03d52899c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6146a8c98cfc4846995b85524ab839b9" is_pre_election: true
I20260928 02:46:16.923488 9528 raft_consensus.cc:2454] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcb6f9f5704469e955955d03d52899c in term 0.
I20260928 02:46:16.923460 9598 leader_election.cc:302] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966; no voters:
I20260928 02:46:16.923590 9861 raft_consensus.cc:2781] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:16.923651 9861 raft_consensus.cc:482] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:16.923681 9861 raft_consensus.cc:3037] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:16.924502 9861 raft_consensus.cc:504] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.924714 9861 leader_election.cc:288] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [CANDIDATE]: Term 1 election: Requested vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387)
I20260928 02:46:16.924795 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a804093c9519465fa8db265885ef97ef" candidate_uuid: "0bcb6f9f5704469e955955d03d52899c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf4b8eab51db4474a6631d696d145966"
I20260928 02:46:16.924822 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a804093c9519465fa8db265885ef97ef" candidate_uuid: "0bcb6f9f5704469e955955d03d52899c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6146a8c98cfc4846995b85524ab839b9"
I20260928 02:46:16.924867 9796 raft_consensus.cc:3037] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:16.924903 9528 raft_consensus.cc:3037] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:16.925454 9796 raft_consensus.cc:2454] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcb6f9f5704469e955955d03d52899c in term 1.
I20260928 02:46:16.925618 9598 leader_election.cc:302] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966; no voters:
I20260928 02:46:16.925802 9528 raft_consensus.cc:2454] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcb6f9f5704469e955955d03d52899c in term 1.
I20260928 02:46:16.925913 9861 raft_consensus.cc:2781] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:16.926046 9861 raft_consensus.cc:686] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 LEADER]: Becoming Leader. State: Replica: 0bcb6f9f5704469e955955d03d52899c, State: Running, Role: LEADER
I20260928 02:46:16.926141 9861 consensus_queue.cc:237] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:16.927109 9382 catalog_manager.cc:5495] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bcb6f9f5704469e955955d03d52899c (127.27.109.130). New cstate: current_term: 1 leader_uuid: "0bcb6f9f5704469e955955d03d52899c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:16.940243 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.937097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.27.109.130:46687)
0928 02:46:16.937134 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:16.937136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:16.937144 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:46:16.937216 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:16.937216 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:16.937230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:16.937324 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:16.937474 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.938090 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.939312 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:16.939630 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:16.939694 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:16.939871 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:16.939984 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:16.940076 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:16.940134 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 02:46:16.940153 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:46:16.941673 9796 raft_consensus.cc:1261] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Refusing update from remote peer 0bcb6f9f5704469e955955d03d52899c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:16.941694 9528 raft_consensus.cc:1261] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Refusing update from remote peer 0bcb6f9f5704469e955955d03d52899c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:16.942008 9866 consensus_queue.cc:1035] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:16.942134 9861 consensus_queue.cc:1035] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:16.944171 9869 mvcc.cc:205] Tried to move back new op lower bound from 7334148411151106048 to 7334148411090124800. Current Snapshot: MvccSnapshot[applied={T|T < 7334148411151106048}]
I20260928 02:46:16.946095 9382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:16.946182 9382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:16.946841 9382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Sending DeleteTablet for 3 replicas of tablet a804093c9519465fa8db265885ef97ef
I20260928 02:46:16.947129 9776 tablet_service.cc:1514] Processing DeleteTablet for tablet a804093c9519465fa8db265885ef97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:16 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45234
I20260928 02:46:16.947190 9508 tablet_service.cc:1514] Processing DeleteTablet for tablet a804093c9519465fa8db265885ef97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:16 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:41988
I20260928 02:46:16.947356 9879 tablet_replica.cc:330] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: stopping tablet replica
I20260928 02:46:16.947391 9642 tablet_service.cc:1514] Processing DeleteTablet for tablet a804093c9519465fa8db265885ef97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:16 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:56504
I20260928 02:46:16.947441 9879 raft_consensus.cc:2229] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:16.947453 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9445
I20260928 02:46:16.947546 9879 raft_consensus.cc:2258] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:16.948096 9879 ts_tablet_manager.cc:1905] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:16.949981 9879 ts_tablet_manager.cc:1918] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:16.950091 9879 log.cc:1196] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/a804093c9519465fa8db265885ef97ef
I20260928 02:46:16.950399 9879 ts_tablet_manager.cc:1939] T a804093c9519465fa8db265885ef97ef P cf4b8eab51db4474a6631d696d145966: Deleting consensus metadata
I20260928 02:46:16.950834 9370 catalog_manager.cc:4841] TS cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): tablet a804093c9519465fa8db265885ef97ef (table test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]) successfully deleted
W20260928 02:46:16.953886 9369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:39387 (error 108)
I20260928 02:46:16.953981 9881 tablet_replica.cc:330] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: stopping tablet replica
I20260928 02:46:16.954067 9881 raft_consensus.cc:2229] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:16.954072 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9579
I20260928 02:46:16.954150 9881 raft_consensus.cc:2258] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:16.955106 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.954962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.0.0.1:41990)
0928 02:46:16.955033 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.955070 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:39387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:46:16.955214 9369 catalog_manager.cc:4584] TS 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a804093c9519465fa8db265885ef97ef: Network error: Client connection negotiation failed: client connection to 127.27.109.129:39387: connect: Connection refused (error 111)
I20260928 02:46:16.959089 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9713
I20260928 02:46:16.959183 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:16.959053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:46687 (local address 127.0.0.1:56534)
0928 02:46:16.959110 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:16.959151 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.130:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:46:16.963439 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9351
I20260928 02:46:16.968441 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--webserver_interface=127.27.109.190
--webserver_port=42565
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:17.047230 9883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.047433 9883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.047454 9883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.048517 9883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:17.048549 9883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.048563 9883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.048576 9883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:17.048588 9883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:17.049943 9883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=42565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.050117 9883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.050276 9883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.058523 9889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.058504 9892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.058641 9890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.058650 9883 server_base.cc:1034] running on GCE node
I20260928 02:46:17.058955 9883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.059217 9883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.060351 9883 hybrid_clock.cc:648] HybridClock initialized: now 1790563577060341 us; error 26 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.062928 9883 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:17.063380 9883 webserver.cc:460] Webserver started at http://127.27.109.190:42565/ using document root <none> and password file <none>
I20260928 02:46:17.063627 9883 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.063705 9883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.065001 9883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:17.065644 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.065835 9883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.065938 9883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "526c5fd853ea4dbeacd46acb147fd2a3"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "b98bbc1e31af99a3247fea798f763966"
server_key_iv: "85187346d86f6a0569fc2ffbb5393b7f"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.066298 9883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.107093 9883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.109460 9883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.109632 9883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.113592 9883 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37383
I20260928 02:46:17.113631 9951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37383 every 8 connection(s)
I20260928 02:46:17.114028 9883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:17.115553 9952 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:17.116147 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap starting.
I20260928 02:46:17.123190 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9883
I20260928 02:46:17.123482 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:39387
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=44433
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:17.124132 9952 log.cc:824] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.126618 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:17.126911 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap complete.
I20260928 02:46:17.130869 9952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.131232 9952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.131410 9952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.131505 9952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:17.131564 9952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:17.131625 9952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:17.133136 9952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.133255 9952 leader_election.cc:302] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526c5fd853ea4dbeacd46acb147fd2a3; no voters:
I20260928 02:46:17.133412 9952 leader_election.cc:288] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:46:17.133450 9956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:46:17.133605 9956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 LEADER]: Becoming Leader. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Running, Role: LEADER
I20260928 02:46:17.133692 9952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:17.133689 9956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.134799 9957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:17.135098 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 526c5fd853ea4dbeacd46acb147fd2a3. Latest consensus state: current_term: 2 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:17.135182 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.134907 9957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.135716 9971 catalog_manager.cc:1259] Loaded cluster ID: 2955d2394050405da50c61f3ace2ea19
I20260928 02:46:17.135764 9971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: loading cluster ID for follower catalog manager: success
W20260928 02:46:17.135950 9971 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 02:46:17.136001 9971 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 02:46:17.136394 9973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:17.136793 9973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:17.136864 9973 catalog_manager.cc:1259] Loaded cluster ID: 2955d2394050405da50c61f3ace2ea19
I20260928 02:46:17.136894 9973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:17.137042 9973 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:46:17.142077 9973 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:46:17.144345 9973 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:17.144522 9973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Loaded TSK: 0
I20260928 02:46:17.144738 9973 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:46:17.207134 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.207325 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.207348 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.208462 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.208499 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.208525 9955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:17.209889 9955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:39387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=44433
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.210096 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.210318 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.215910 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.215986 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.216001 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.216286 9955 server_base.cc:1034] running on GCE node
I20260928 02:46:17.216482 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.216728 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.217906 9955 hybrid_clock.cc:648] HybridClock initialized: now 1790563577217887 us; error 34 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.220670 9955 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:17.221091 9955 webserver.cc:460] Webserver started at http://127.27.109.129:44433/ using document root <none> and password file <none>
I20260928 02:46:17.221318 9955 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.221398 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.222716 9955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:17.223469 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.223641 9955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:17.223737 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6146a8c98cfc4846995b85524ab839b9"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "ac1eda41151054561e7c419a6f981734"
server_key_iv: "bf852c2585bc90eb97061d7f49676b45"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.224046 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.248989 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.251089 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.251248 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.251485 9955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:17.251905 9996 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:46:17.253680 9955 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:46:17.253742 9955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:46:17.253788 9955 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:46:17.254392 9955 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:46:17.254458 9955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:17.254508 9996 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Bootstrap starting.
I20260928 02:46:17.261315 9955 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:39387
I20260928 02:46:17.261427 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:39387 every 8 connection(s)
I20260928 02:46:17.261725 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:17.264928 9996 log.cc:824] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.267232 9996 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:17.267534 9996 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Bootstrap complete.
I20260928 02:46:17.268455 9996 ts_tablet_manager.cc:1397] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:17.269091 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 9955
I20260928 02:46:17.269364 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:46687
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=39951
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:17.270043 9996 raft_consensus.cc:348] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:17.270416 9996 raft_consensus.cc:729] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6146a8c98cfc4846995b85524ab839b9, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.270573 9996 consensus_queue.cc:260] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:17.270870 9996 ts_tablet_manager.cc:1428] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260928 02:46:17.277130 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.262366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:45471)
0928 02:46:17.262581 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.262623 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.263071 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.263595 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.263598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.263911 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:17.264098 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.264103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.265714 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.265715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.266537 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.266539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.267328 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.267329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.267464 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.268017 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:17.268025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:17.269430 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:17.275638 (+ 6208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.275644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.275645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.275866 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.276152 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.276152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.276152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.276244 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.276701 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:17.276705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:17.276806 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.276883 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.276998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":140,"threads_started":1}
I20260928 02:46:17.277350 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.262500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:45471 (local address 127.27.109.190:37383)
0928 02:46:17.262724 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.262737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.262794 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:46:17.263313 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.263322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.263393 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.263590 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.264237 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.265602 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.266620 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.267195 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.267337 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.269561 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.269565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.269566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.269577 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.275557 (+ 5980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.275951 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.275954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.275955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.276078 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.276309 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.276310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.276310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.276628 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.276723 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.276970 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.277121 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260928 02:46:17.277909 10104 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:17.277999 10104 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:17.278175 10104 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:17.278851 9916 ts_manager.cc:194] Registered new tserver with Master: 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387)
I20260928 02:46:17.279963 9916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:45471
I20260928 02:46:17.281049 10104 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:46:17.287225 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.279372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.0.0.1:42006)
0928 02:46:17.279547 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.279590 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.279662 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.279839 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.279843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.280146 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:17.280199 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.280201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.281136 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.281137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.281947 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.281952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.282494 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.282495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.282571 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.283062 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:17.283066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:17.284450 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:17.286051 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.286057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.286059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.286250 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.286506 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.286507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.286507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.286568 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.286854 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:17.286860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:17.286908 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.287009 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.287127 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260928 02:46:17.287223 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.279558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.27.109.129:39387)
0928 02:46:17.279686 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.279690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.279702 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:17.279734 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.279736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.279750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.279824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.280269 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.281006 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.282068 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.282386 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.282511 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.284550 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.284557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.284557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.284564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.285950 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.286342 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.286345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.286345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.286409 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.286638 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.286639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.286639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.286782 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.286875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.287044 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.287132 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260928 02:46:17.287599 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet a804093c9519465fa8db265885ef97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:16 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:42006
I20260928 02:46:17.288092 10113 tablet_replica.cc:330] stopping tablet replica
I20260928 02:46:17.288197 10113 raft_consensus.cc:2229] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:17.288261 10113 raft_consensus.cc:2258] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:17.288564 10113 ts_tablet_manager.cc:1905] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:17.290683 10113 ts_tablet_manager.cc:1918] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:17.290756 10113 log.cc:1196] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/a804093c9519465fa8db265885ef97ef
I20260928 02:46:17.290987 10113 ts_tablet_manager.cc:1939] T a804093c9519465fa8db265885ef97ef P 6146a8c98cfc4846995b85524ab839b9: Deleting consensus metadata
I20260928 02:46:17.291469 9902 catalog_manager.cc:4841] TS 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387): tablet a804093c9519465fa8db265885ef97ef (table test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]) successfully deleted
W20260928 02:46:17.352613 10109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.352815 10109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.352838 10109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.353986 10109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.354027 10109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.354053 10109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:17.355542 10109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:46687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=39951
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.355764 10109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.355985 10109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.360677 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.360728 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.360924 10109 server_base.cc:1034] running on GCE node
W20260928 02:46:17.360678 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.361163 10109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.361342 10109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.362528 10109 hybrid_clock.cc:648] HybridClock initialized: now 1790563577362513 us; error 27 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.364881 10109 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:17.365269 10109 webserver.cc:460] Webserver started at http://127.27.109.130:39951/ using document root <none> and password file <none>
I20260928 02:46:17.365490 10109 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.365571 10109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.366875 10109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:17.367540 10128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.367703 10109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:17.367797 10109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "0bcb6f9f5704469e955955d03d52899c"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "834bf54a85c62bf63e35330ab3264aea"
server_key_iv: "f205f3bede9a8158177e3cbe1c0a2eb5"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.368094 10109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.377344 10109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.379495 10109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.379643 10109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.379853 10109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:17.380244 10135 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:46:17.381640 10109 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:46:17.381696 10109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:46:17.381758 10109 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:46:17.382392 10109 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:46:17.382449 10109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:17.382527 10135 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Bootstrap starting.
I20260928 02:46:17.388767 10109 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:46687
I20260928 02:46:17.388850 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:46687 every 8 connection(s)
I20260928 02:46:17.389153 10109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:17.392761 10135 log.cc:824] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.394181 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10109
I20260928 02:46:17.394493 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:42595
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=40281
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:17.395411 10135 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:17.395715 10135 tablet_bootstrap.cc:492] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Bootstrap complete.
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:17.396891 10135 ts_tablet_manager.cc:1397] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 02:46:17.398538 10135 raft_consensus.cc:348] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:17.398926 10135 raft_consensus.cc:729] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb6f9f5704469e955955d03d52899c, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.399084 10135 consensus_queue.cc:260] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } }
I20260928 02:46:17.399317 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.389847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:59445 (local address 127.27.109.190:37383)
0928 02:46:17.389918 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.389921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.390111 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:46:17.390933 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.390934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.390948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.390998 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.391942 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.392423 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.393826 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.394038 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.394124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.396380 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.396382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.396382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.396389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.397874 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.398283 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.398285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.398285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.398346 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.398657 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.398658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.398658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.398775 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.398851 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.399140 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.399228 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:46:17.399353 10135 ts_tablet_manager.cc:1428] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260928 02:46:17.399524 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.389795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:59445)
0928 02:46:17.389986 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.390032 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.390828 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.391076 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.391080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.391562 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:17.391838 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.391848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.392543 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.392544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.393737 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.393740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.394148 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.394149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.394300 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.394824 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:17.394832 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:17.396282 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:17.397981 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.397988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.397990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.398208 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.398419 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.398419 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.398420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.398587 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.398843 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:17.398849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:17.399009 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.399108 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.399259 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
I20260928 02:46:17.400070 10243 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:17.400156 10243 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:17.400308 10243 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:17.400818 9916 ts_manager.cc:194] Registered new tserver with Master: 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
I20260928 02:46:17.401451 9916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:59445
I20260928 02:46:17.402311 10243 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.130@KRBTEST.COM
I20260928 02:46:17.409188 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.401144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:46687 (local address 127.0.0.1:56542)
0928 02:46:17.401194 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.401204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.401240 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.401474 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.401475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.401636 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:17.401675 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.401677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.402568 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.402570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.403451 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.403454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.403955 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.403956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.404041 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.404373 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:17.404378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:17.405620 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:17.407883 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.407884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.407884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.408163 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.408388 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.408389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.408390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.408490 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.408835 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:17.408837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:17.408878 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.408961 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.409072 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:46:17.409607 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.401198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.27.109.130:46687)
0928 02:46:17.401334 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.401338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.401346 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:46:17.401381 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.401382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.401394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.401474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.401755 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.402465 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.403529 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.403879 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.403949 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.405704 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.405712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.405712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.405718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.407786 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.408237 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.408239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.408240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.408311 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.408575 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.408576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.408577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.408756 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.408833 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.409413 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.409515 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260928 02:46:17.409889 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet a804093c9519465fa8db265885ef97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:16 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:56542
I20260928 02:46:17.410421 10250 tablet_replica.cc:330] stopping tablet replica
I20260928 02:46:17.410539 10250 raft_consensus.cc:2229] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:17.410605 10250 raft_consensus.cc:2258] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:17.411623 10250 ts_tablet_manager.cc:1905] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:17.413731 10250 ts_tablet_manager.cc:1918] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:17.413803 10250 log.cc:1196] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/a804093c9519465fa8db265885ef97ef
I20260928 02:46:17.414040 10250 ts_tablet_manager.cc:1939] T a804093c9519465fa8db265885ef97ef P 0bcb6f9f5704469e955955d03d52899c: Deleting consensus metadata
I20260928 02:46:17.414575 9900 catalog_manager.cc:4841] TS 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): tablet a804093c9519465fa8db265885ef97ef (table test-table [id=f72f9cfa9e7549789b3203cf1a132cb4]) successfully deleted
W20260928 02:46:17.489827 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.490115 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.490163 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.492226 10247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.492300 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.492347 10247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:17.494853 10247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:42595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=40281
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.495117 10247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.495390 10247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.500771 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.500758 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.500777 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.501041 10247 server_base.cc:1034] running on GCE node
I20260928 02:46:17.501242 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.501502 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.502668 10247 hybrid_clock.cc:648] HybridClock initialized: now 1790563577502654 us; error 33 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.506323 10247 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:17.506858 10247 webserver.cc:460] Webserver started at http://127.27.109.131:40281/ using document root <none> and password file <none>
I20260928 02:46:17.507092 10247 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.507162 10247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.508735 10247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:17.509323 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.509510 10247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:17.509626 10247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "cf4b8eab51db4474a6631d696d145966"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "01ef11aaa8314676c7b86156943524a5"
server_key_iv: "95214030b5f0b3f087dbb276beff0892"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.510075 10247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.536041 10247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.538393 10247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.538575 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.538820 10247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:17.539140 10247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:17.539184 10247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.539218 10247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:17.539248 10247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.546511 10247 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:42595
I20260928 02:46:17.546566 10378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:42595 every 8 connection(s)
I20260928 02:46:17.546947 10247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:17.551101 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10247
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:17.557204 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.547673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:40393 (local address 127.27.109.190:37383)
0928 02:46:17.547743 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.547746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.547901 (+ 155us) server_negotiation.cc:407] Connection header received
0928 02:46:17.548423 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.548424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.548437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.548520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.549414 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.550241 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.551264 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.551564 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.551612 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.554239 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.554240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.554240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.554248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.555744 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.556190 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.556191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.556191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.556241 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.556524 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.556525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.556525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.556658 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.556725 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.557042 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.557104 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:46:17.557413 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.547634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:40393)
0928 02:46:17.547787 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.547826 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.548331 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.548579 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.548583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.549147 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:17.549325 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.549329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.550329 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.550330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.551137 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.551139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.551708 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.551708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.551809 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.552313 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:17.552321 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:17.554148 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:17.555802 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.555808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.555810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.556119 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.556300 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:17.556301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:17.556302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:17.556436 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:17.556727 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:17.556733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:17.556895 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.557021 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.557176 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 02:46:17.557850 10379 heartbeater.cc:347] Connected to a master server at 127.27.109.190:37383
I20260928 02:46:17.557947 10379 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:17.558090 10379 heartbeater.cc:511] Master 127.27.109.190:37383 requested a full tablet report, sending...
I20260928 02:46:17.558429 9916 ts_manager.cc:194] Registered new tserver with Master: cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595)
I20260928 02:46:17.558915 9916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:40393
I20260928 02:46:17.559122 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.551555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.27.109.190:37383)
0928 02:46:17.551663 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.551665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.551671 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:46:17.551698 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.551699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.551708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.551789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:17.552501 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.553170 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.553975 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.554181 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.554255 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.555736 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:17.555738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:17.555738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:17.555746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:17.557817 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.558213 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.558214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.558215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.558273 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:17.558513 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:17.558514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:17.558514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:17.558635 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:17.558737 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.558842 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.558901 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260928 02:46:17.559827 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:17.562064 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.560115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.27.109.190:37383)
0928 02:46:17.560147 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.560148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.560239 (+ 91us) server_negotiation.cc:407] Connection header received
0928 02:46:17.560290 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.560291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.560310 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.560368 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:17.560439 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.560852 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.561517 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.561689 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.561736 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.561853 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:17.561928 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:17.561977 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.562011 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.562025 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:46:17.562294 9916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:17.563894 9916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:17.565991 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 (DEFAULT_TABLE table=test-table [id=7c8ad00eb8bd42edab804f4b17616c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:17.566059 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 (DEFAULT_TABLE table=test-table [id=7c8ad00eb8bd42edab804f4b17616c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:17.566181 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b3057cca54ddb9f8397f3f6732c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:17.566267 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b3057cca54ddb9f8397f3f6732c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:17.567575 9996 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Bootstrap starting.
I20260928 02:46:17.567756 10135 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Bootstrap starting.
I20260928 02:46:17.567905 9996 tablet_bootstrap.cc:654] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:17.568001 10135 tablet_bootstrap.cc:654] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:17.568684 9996 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: No bootstrap required, opened a new log
I20260928 02:46:17.568689 10135 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: No bootstrap required, opened a new log
I20260928 02:46:17.568735 9996 ts_tablet_manager.cc:1397] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:17.568735 10135 ts_tablet_manager.cc:1397] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:17.568881 10135 raft_consensus.cc:348] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.568881 9996 raft_consensus.cc:348] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.568929 10135 raft_consensus.cc:374] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:17.568930 9996 raft_consensus.cc:374] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:17.568957 10135 raft_consensus.cc:729] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb6f9f5704469e955955d03d52899c, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.568959 9996 raft_consensus.cc:729] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6146a8c98cfc4846995b85524ab839b9, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.569003 10135 consensus_queue.cc:260] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.569024 9996 consensus_queue.cc:260] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.569100 10135 ts_tablet_manager.cc:1428] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.569128 9996 ts_tablet_manager.cc:1428] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.572445 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.565806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:42595 (local address 127.0.0.1:45246)
0928 02:46:17.565918 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.565930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.565969 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.569388 (+ 3419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.569389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.569390 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:17.569427 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.569429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.570544 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.570545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.571544 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.571547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.572207 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.572208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.572272 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.572289 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.572346 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.572382 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:46:17.572476 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.567607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.27.109.131:42595)
0928 02:46:17.568468 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.568473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.568485 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:17.569254 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.569257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.569290 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.569387 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:17.569489 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.570412 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.571636 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.572110 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.572198 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.572251 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.572361 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.572397 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":80,"threads_started":1}
I20260928 02:46:17.573057 10313 tablet_service.cc:1467] Processing CreateTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 (DEFAULT_TABLE table=test-table [id=7c8ad00eb8bd42edab804f4b17616c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:17.573342 10313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153b3057cca54ddb9f8397f3f6732c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:17.576041 10388 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Bootstrap starting.
I20260928 02:46:17.576620 10388 tablet_bootstrap.cc:654] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:17.577119 10388 log.cc:824] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.577864 10388 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: No bootstrap required, opened a new log
I20260928 02:46:17.577955 10388 ts_tablet_manager.cc:1397] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:17.579325 10388 raft_consensus.cc:348] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.579481 10388 raft_consensus.cc:374] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:17.579527 10388 raft_consensus.cc:729] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4b8eab51db4474a6631d696d145966, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.579664 10388 consensus_queue.cc:260] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.579903 10388 ts_tablet_manager.cc:1428] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:17.579957 10379 heartbeater.cc:503] Master 127.27.109.190:37383 was elected leader, sending a full tablet report...
W20260928 02:46:17.639667 10244 tablet.cc:2369] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:17.696417 10390 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:17.696578 10390 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.697238 10390 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
I20260928 02:46:17.700474 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.697180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:46687 (local address 127.27.109.131:49471)
0928 02:46:17.697384 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.697397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.697444 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.697845 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.697846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.697848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:17.697918 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.697920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.698690 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.698691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.699636 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.699639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.700231 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.700232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.700304 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.700308 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.700368 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.700406 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260928 02:46:17.700475 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.697367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:49471 (local address 127.27.109.130:46687)
0928 02:46:17.697427 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.697429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.697441 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:46:17.697679 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.697680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.697708 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.697780 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:17.698000 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.698609 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.699731 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.700145 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.700234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.700311 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.700374 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.700413 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:46:17.700774 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.697591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:42981 (local address 127.27.109.129:39387)
0928 02:46:17.697633 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.697634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.697700 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:46:17.697792 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.697794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.697819 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.697877 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:17.698078 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.699039 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.700020 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.700419 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.700505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.700572 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.700663 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.700709 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:46:17.700763 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b3057cca54ddb9f8397f3f6732c44" candidate_uuid: "cf4b8eab51db4474a6631d696d145966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb6f9f5704469e955955d03d52899c" is_pre_election: true
I20260928 02:46:17.700906 10197 raft_consensus.cc:2454] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4b8eab51db4474a6631d696d145966 in term 0.
I20260928 02:46:17.700999 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.697204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.27.109.131:42981)
0928 02:46:17.697628 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.697639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:17.697730 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:17.697919 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:17.697920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:17.697921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:17.697977 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.697979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.699137 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.699138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.699925 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:17.699927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.700503 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.700504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:17.700570 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.700574 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:17.700878 (+ 304us) client_negotiation.cc:240] Negotiation successful
0928 02:46:17.700928 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":151,"threads_started":1}
I20260928 02:46:17.701164 10266 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966; no voters:
I20260928 02:46:17.701283 10390 raft_consensus.cc:2781] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:17.701337 10390 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:17.701370 10390 raft_consensus.cc:3037] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:17.701329 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b3057cca54ddb9f8397f3f6732c44" candidate_uuid: "cf4b8eab51db4474a6631d696d145966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6146a8c98cfc4846995b85524ab839b9" is_pre_election: true
I20260928 02:46:17.701478 10058 raft_consensus.cc:2454] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf4b8eab51db4474a6631d696d145966 in term 0.
I20260928 02:46:17.702145 10390 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.702278 10390 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [CANDIDATE]: Term 1 election: Requested vote from peers 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
I20260928 02:46:17.702456 10197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b3057cca54ddb9f8397f3f6732c44" candidate_uuid: "cf4b8eab51db4474a6631d696d145966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb6f9f5704469e955955d03d52899c"
I20260928 02:46:17.702502 10058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153b3057cca54ddb9f8397f3f6732c44" candidate_uuid: "cf4b8eab51db4474a6631d696d145966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6146a8c98cfc4846995b85524ab839b9"
I20260928 02:46:17.702577 10197 raft_consensus.cc:3037] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:17.702592 10058 raft_consensus.cc:3037] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:17.703161 10197 raft_consensus.cc:2454] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4b8eab51db4474a6631d696d145966 in term 1.
I20260928 02:46:17.703294 10266 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966; no voters:
I20260928 02:46:17.703343 10058 raft_consensus.cc:2454] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf4b8eab51db4474a6631d696d145966 in term 1.
I20260928 02:46:17.703374 10390 raft_consensus.cc:2781] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:17.703518 10390 raft_consensus.cc:686] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 1 LEADER]: Becoming Leader. State: Replica: cf4b8eab51db4474a6631d696d145966, State: Running, Role: LEADER
I20260928 02:46:17.703608 10390 consensus_queue.cc:237] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:17.704389 9916 catalog_manager.cc:5495] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf4b8eab51db4474a6631d696d145966 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "cf4b8eab51db4474a6631d696d145966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:17.712755 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.710879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.27.109.131:42595)
0928 02:46:17.710914 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:17.710916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:17.710952 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:46:17.710975 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:17.710975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:17.710989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:17.711037 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:17.711124 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.711533 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.712166 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:17.712340 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:17.712394 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:17.712510 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:17.712592 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:17.712648 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:17.712682 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 02:46:17.712697 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:46:17.714375 10058 raft_consensus.cc:1261] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Refusing update from remote peer cf4b8eab51db4474a6631d696d145966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:17.714375 10197 raft_consensus.cc:1261] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Refusing update from remote peer cf4b8eab51db4474a6631d696d145966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:17.714716 10392 consensus_queue.cc:1035] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:17.714855 10390 consensus_queue.cc:1035] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:17.716647 10395 mvcc.cc:205] Tried to move back new op lower bound from 7334148414315638784 to 7334148414274592768. Current Snapshot: MvccSnapshot[applied={T|T < 7334148414315638784}]
I20260928 02:46:17.719226 9915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:17.719301 9915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:17.719987 9915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Sending DeleteTablet for 3 replicas of tablet 153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:17.720321 10177 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:17 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:56542
I20260928 02:46:17.720350 10313 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:17 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:45246
I20260928 02:46:17.720408 10250 tablet_replica.cc:330] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: stopping tablet replica
I20260928 02:46:17.720455 10250 raft_consensus.cc:2229] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:17.720453 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet 153b3057cca54ddb9f8397f3f6732c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:17 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:42006
I20260928 02:46:17.720510 10250 raft_consensus.cc:2258] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:17.720552 10113 tablet_replica.cc:330] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: stopping tablet replica
I20260928 02:46:17.720607 10113 raft_consensus.cc:2229] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:17.720645 10113 raft_consensus.cc:2258] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:17.720678 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9955
I20260928 02:46:17.723315 10250 ts_tablet_manager.cc:1905] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:17.724820 10250 ts_tablet_manager.cc:1918] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:17.724884 10250 log.cc:1196] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:17.725095 10250 ts_tablet_manager.cc:1939] T 153b3057cca54ddb9f8397f3f6732c44 P 0bcb6f9f5704469e955955d03d52899c: Deleting consensus metadata
I20260928 02:46:17.725452 9900 catalog_manager.cc:4841] TS 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): tablet 153b3057cca54ddb9f8397f3f6732c44 (table test-table [id=7c8ad00eb8bd42edab804f4b17616c5c]) successfully deleted
W20260928 02:46:17.726534 9902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:39387 (error 108)
I20260928 02:46:17.726743 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10109
I20260928 02:46:17.726758 10404 tablet_replica.cc:330] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: stopping tablet replica
I20260928 02:46:17.726841 10404 raft_consensus.cc:2229] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:17.726925 10404 raft_consensus.cc:2258] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:17.727355 10404 ts_tablet_manager.cc:1905] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:17.729230 10404 ts_tablet_manager.cc:1918] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:17.729290 10404 log.cc:1196] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:17.729565 10404 ts_tablet_manager.cc:1939] T 153b3057cca54ddb9f8397f3f6732c44 P cf4b8eab51db4474a6631d696d145966: Deleting consensus metadata
I20260928 02:46:17.729956 9903 catalog_manager.cc:4841] TS cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): tablet 153b3057cca54ddb9f8397f3f6732c44 (table test-table [id=7c8ad00eb8bd42edab804f4b17616c5c]) successfully deleted
I20260928 02:46:17.731001 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:17.730852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:39387 (local address 127.0.0.1:42012)
0928 02:46:17.730912 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:17.730958 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:39387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 02:46:17.731165 9902 catalog_manager.cc:4584] TS 6146a8c98cfc4846995b85524ab839b9 (127.27.109.129:39387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 153b3057cca54ddb9f8397f3f6732c44: Network error: Client connection negotiation failed: client connection to 127.27.109.129:39387: connect: Connection refused (error 111)
I20260928 02:46:17.733305 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10247
I20260928 02:46:17.738440 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 9883
I20260928 02:46:17.743945 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--webserver_interface=127.27.109.190
--webserver_port=42565
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:17.821854 10406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.822041 10406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.822062 10406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.823261 10406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:17.823303 10406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.823318 10406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.823333 10406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:17.823345 10406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:17.824810 10406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=42565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.825045 10406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.825263 10406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.829831 10411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.829811 10412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.830008 10414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.829908 10406 server_base.cc:1034] running on GCE node
I20260928 02:46:17.830317 10406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.830608 10406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.831746 10406 hybrid_clock.cc:648] HybridClock initialized: now 1790563577831734 us; error 29 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.834100 10406 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:17.834539 10406 webserver.cc:460] Webserver started at http://127.27.109.190:42565/ using document root <none> and password file <none>
I20260928 02:46:17.834756 10406 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.834800 10406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.836045 10406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:17.836755 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.836951 10406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:17.837021 10406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "526c5fd853ea4dbeacd46acb147fd2a3"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "b98bbc1e31af99a3247fea798f763966"
server_key_iv: "85187346d86f6a0569fc2ffbb5393b7f"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.837338 10406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.847886 10406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.849625 10406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.849757 10406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.853475 10406 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37383
I20260928 02:46:17.853509 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37383 every 8 connection(s)
I20260928 02:46:17.853864 10406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:17.855427 10474 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:17.856055 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap starting.
I20260928 02:46:17.858270 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10406
I20260928 02:46:17.858579 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:39387
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=44433
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:17.864776 10474 log.cc:824] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:17.867038 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:17.867352 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3: Bootstrap complete.
I20260928 02:46:17.869535 10474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.869776 10474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Initialized, Role: FOLLOWER
I20260928 02:46:17.869989 10474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.870090 10474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:17.870142 10474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:17.870191 10474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:46:17.871215 10474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.871330 10474 leader_election.cc:302] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 526c5fd853ea4dbeacd46acb147fd2a3; no voters:
I20260928 02:46:17.871524 10474 leader_election.cc:288] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:46:17.871555 10478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:46:17.871665 10478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [term 3 LEADER]: Becoming Leader. State: Replica: 526c5fd853ea4dbeacd46acb147fd2a3, State: Running, Role: LEADER
I20260928 02:46:17.871757 10478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } }
I20260928 02:46:17.871788 10474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:17.872023 10479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:17.872088 10478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 526c5fd853ea4dbeacd46acb147fd2a3. Latest consensus state: current_term: 3 leader_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "526c5fd853ea4dbeacd46acb147fd2a3" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37383 } } }
I20260928 02:46:17.872254 10478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.872578 10479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 526c5fd853ea4dbeacd46acb147fd2a3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:17.872632 10484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:17.873520 10484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:17.873654 10484 catalog_manager.cc:1259] Loaded cluster ID: 2955d2394050405da50c61f3ace2ea19
I20260928 02:46:17.873929 10484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:46:17.874141 10484 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f19b5a3b9ad google::LogMessage::Fail() at ??:0
@ 0x7f19b5a3e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f19b5a3b4de google::LogMessage::Flush() at ??:0
@ 0x7f19b5a3ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f19bd95c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f19bd9658b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f19b79090d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f19b78fe954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f19b87626db start_thread at ??:0
@ 0x7f19b453961f clone at ??:0
W20260928 02:46:17.953063 10476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:17.953239 10476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:17.953261 10476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:17.954411 10476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:17.954450 10476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:17.954535 10476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:17.955946 10476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:39387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=44433
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:17.956199 10476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:17.956421 10476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:17.961751 10511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.961807 10513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:17.961860 10510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:17.962174 10476 server_base.cc:1034] running on GCE node
I20260928 02:46:17.962383 10476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:17.962661 10476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:17.963866 10476 hybrid_clock.cc:648] HybridClock initialized: now 1790563577963853 us; error 27 us; skew 500 ppm
Sep 28 02:46:17 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:17.966923 10476 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:17.967520 10476 webserver.cc:460] Webserver started at http://127.27.109.129:44433/ using document root <none> and password file <none>
I20260928 02:46:17.967792 10476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:17.967870 10476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:17.969660 10476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 02:46:17.970453 10520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:17.970664 10476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:17.970744 10476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "6146a8c98cfc4846995b85524ab839b9"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "ac1eda41151054561e7c419a6f981734"
server_key_iv: "bf852c2585bc90eb97061d7f49676b45"
server_key_version: "encryptionkey@0"
I20260928 02:46:17.971077 10476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:17.983750 10476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:17.985947 10476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:17.986078 10476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:17.986301 10476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:17.986779 10527 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:46:17.988816 10476 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:46:17.988879 10476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:46:17.988920 10476 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:46:17.989507 10476 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:46:17.989560 10476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:17.989620 10527 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Bootstrap starting.
I20260928 02:46:17.996886 10476 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:39387
I20260928 02:46:17.997243 10476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:18.000746 10527 log.cc:824] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:18.001672 10634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:39387 every 8 connection(s)
I20260928 02:46:18.003567 10527 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:18.003904 10527 tablet_bootstrap.cc:492] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Bootstrap complete.
I20260928 02:46:18.003926 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10476
I20260928 02:46:18.004220 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:46687
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=39951
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:18.005214 10527 ts_tablet_manager.cc:1397] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260928 02:46:18.006867 10527 raft_consensus.cc:348] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:18.007200 10527 raft_consensus.cc:729] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6146a8c98cfc4846995b85524ab839b9, State: Initialized, Role: FOLLOWER
I20260928 02:46:18.007390 10527 consensus_queue.cc:260] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:18.007688 10527 ts_tablet_manager.cc:1428] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 02:46:18.082401 10638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:46:18.001381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:45979)
0928 02:46:18.001538 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.001580 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:18.002221 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:18.082221 (+ 80000us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
W20260928 02:46:18.082719 10635 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37383 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:37383: Client connection negotiation failed: client connection to 127.27.109.190:37383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:46:18.083133 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.083008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:41131)
0928 02:46:18.083063 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.083103 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:46:18.083457 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.083365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:33541)
0928 02:46:18.083411 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.083433 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:46:18.083577 10635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:46:18.088860 10639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:18.089059 10639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:18.089108 10639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:18.090233 10639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:18.090296 10639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:18.090342 10639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:18.091861 10639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:46687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=39951
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:18.092123 10639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:18.092350 10639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:18.097330 10648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:18.097440 10639 server_base.cc:1034] running on GCE node
W20260928 02:46:18.097323 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:18.097321 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:18.097735 10639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:18.097921 10639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:18.099049 10639 hybrid_clock.cc:648] HybridClock initialized: now 1790563578099030 us; error 34 us; skew 500 ppm
Sep 28 02:46:18 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563578, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:18.101444 10639 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:18.101850 10639 webserver.cc:460] Webserver started at http://127.27.109.130:39951/ using document root <none> and password file <none>
I20260928 02:46:18.102083 10639 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:18.102128 10639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:18.103428 10639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:18.103972 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.104189 10639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:18.104339 10639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "0bcb6f9f5704469e955955d03d52899c"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "834bf54a85c62bf63e35330ab3264aea"
server_key_iv: "f205f3bede9a8158177e3cbe1c0a2eb5"
server_key_version: "encryptionkey@0"
I20260928 02:46:18.104638 10639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:18.138355 10639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:18.140281 10639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:18.140455 10639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:18.140691 10639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:18.141008 10639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:18.141067 10639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.141101 10639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:18.141130 10639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.146938 10639 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:46687
I20260928 02:46:18.146991 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:46687 every 8 connection(s)
I20260928 02:46:18.147328 10639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:18.148301 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.147908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:57571)
0928 02:46:18.148089 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.148162 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
W20260928 02:46:18.148672 10769 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37383 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:37383: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
I20260928 02:46:18.148950 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.148843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:49845)
0928 02:46:18.148895 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.148928 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:18.149451 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.149346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:37123)
0928 02:46:18.149397 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.149430 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:18.149590 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10639
W20260928 02:46:18.149657 10769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:46:18.149873 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:42595
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:37383
--webserver_port=40281
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:18.229893 10772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:18.230079 10772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:18.230103 10772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:18.231262 10772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:18.231302 10772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:18.231355 10772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:18.232833 10772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:42595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=40281
--tserver_master_addrs=127.27.109.190:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:18.233100 10772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:18.233326 10772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:18.238229 10778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:18.238229 10777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:18.238317 10780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:18.238408 10772 server_base.cc:1034] running on GCE node
I20260928 02:46:18.238667 10772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:18.238919 10772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:18.240087 10772 hybrid_clock.cc:648] HybridClock initialized: now 1790563578240074 us; error 43 us; skew 500 ppm
Sep 28 02:46:18 dist-test-slave-b5p5 krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563578, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:18.242431 10772 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:18.242877 10772 webserver.cc:460] Webserver started at http://127.27.109.131:40281/ using document root <none> and password file <none>
I20260928 02:46:18.243119 10772 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:18.243191 10772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:18.244504 10772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:18.245048 10787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.245199 10772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:18.245296 10772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "cf4b8eab51db4474a6631d696d145966"
format_stamp: "Formatted at 2026-09-28 02:46:16 on dist-test-slave-b5p5"
server_key: "01ef11aaa8314676c7b86156943524a5"
server_key_iv: "95214030b5f0b3f087dbb276beff0892"
server_key_version: "encryptionkey@0"
I20260928 02:46:18.245637 10772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:18.261636 10772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:18.263469 10772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:18.263623 10772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:18.263834 10772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:18.264123 10772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:18.264178 10772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.264218 10772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:18.264246 10772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:18.269853 10772 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:42595
I20260928 02:46:18.269891 10900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:42595 every 8 connection(s)
I20260928 02:46:18.270205 10772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:46:18.271135 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.270800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:41671)
0928 02:46:18.270938 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.271012 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
W20260928 02:46:18.271515 10901 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:37383 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:37383: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
I20260928 02:46:18.271803 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.271719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:38373)
0928 02:46:18.271763 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.271787 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:46:18.272082 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:18.272009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:33081)
0928 02:46:18.272047 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:18.272067 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:46:18.272208 10901 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:46:18.274576 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10772
I20260928 02:46:19.084096 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.083870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:35323)
0928 02:46:19.083990 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.084045 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 02:46:19.150238 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.150000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:41537)
0928 02:46:19.150145 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.150174 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260928 02:46:19.272679 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.272477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:45953)
0928 02:46:19.272602 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.272635 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260928 02:46:19.791221 10908 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:19.791337 10908 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:19.791610 10908 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
Sep 28 02:46:19 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.131@KRBTEST.COM
Sep 28 02:46:19 dist-test-slave-b5p5 krb5kdc[9338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563577, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.130@KRBTEST.COM
I20260928 02:46:19.802280 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.791878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:50051 (local address 127.27.109.131:42595)
0928 02:46:19.792047 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:19.792055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:19.792081 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:46:19.792649 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:19.792652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:19.792710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:19.792863 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:19.793601 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.794425 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.795587 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.795963 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.796077 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.798274 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:19.798278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:19.798279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:19.798290 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:19.800411 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.800891 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.800894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.800894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.801012 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.801303 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.801304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.801304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.801608 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:19.801695 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:19.802043 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:46:19.802168 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":51,"threads_started":1}
I20260928 02:46:19.802280 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.791896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:60955 (local address 127.27.109.130:46687)
0928 02:46:19.792041 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:19.792055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:19.792081 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:46:19.792649 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:19.792652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:19.792710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:19.792863 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:19.793563 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.794424 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.795553 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.795963 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.796097 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.798836 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:19.798840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:19.798840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:19.798851 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:19.800620 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.800983 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.800985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.800986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.801085 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:19.801303 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:19.801303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:19.801304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:19.801613 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:19.801695 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:19.802038 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:46:19.802168 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260928 02:46:19.802290 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.791764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:46687 (local address 127.27.109.129:60955)
0928 02:46:19.792032 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.792040 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:19.792080 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:19.792915 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:19.792917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:19.793238 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:19.793453 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.793458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.794603 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.794604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.795483 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.795486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.796066 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.796067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.796168 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.796734 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:19.796741 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:19.798719 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:19.800695 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.800696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.800697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.800916 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.801148 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.801149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.801149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.801230 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.801686 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:19.801692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:19.801841 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:46:19.801969 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:46:19.802161 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":93,"thread_start_us":49,"threads_started":1}
I20260928 02:46:19.802317 10909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:19.791763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:42595 (local address 127.27.109.129:50051)
0928 02:46:19.791895 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:19.791907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:19.791964 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:19.792987 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:19.792995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:19.793238 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:19.793453 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.793458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.794571 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.794573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.795487 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:19.795490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:19.796045 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:19.796047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:19.796168 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:19.796782 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:19.796787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:19.798170 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:19.800494 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.800500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.800501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.800811 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.801087 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:19.801088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:19.801088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:19.801230 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:19.801686 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:19.801692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:19.801841 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:46:19.801969 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:46:19.802216 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":108,"thread_start_us":63,"threads_started":1}
W20260928 02:46:19.803287 10521 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
W20260928 02:46:19.803413 10524 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:19.803478 10524 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6146a8c98cfc4846995b85524ab839b9; no voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966
I20260928 02:46:19.803618 10908 raft_consensus.cc:2726] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:46:20.084683 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.084550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:49163)
0928 02:46:20.084605 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:20.084644 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:46:20.150873 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.150644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:56711)
0928 02:46:20.150783 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:20.150829 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260928 02:46:20.273260 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:20.273056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:36295)
0928 02:46:20.273177 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:20.273223 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260928 02:46:21.085307 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.085098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:39915)
0928 02:46:21.085229 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:21.085262 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 02:46:21.151505 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.151323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:53579)
0928 02:46:21.151436 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:21.151468 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 02:46:21.273867 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:21.273691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:50103)
0928 02:46:21.273800 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:21.273832 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260928 02:46:21.853515 10918 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:21.853600 10918 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:21.853726 10918 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
W20260928 02:46:21.854152 10521 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
W20260928 02:46:21.854233 10524 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:21.854275 10524 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6146a8c98cfc4846995b85524ab839b9; no voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966
I20260928 02:46:21.854368 10918 raft_consensus.cc:2726] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:46:22.086030 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.085814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:55947)
0928 02:46:22.085946 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:22.085982 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260928 02:46:22.152148 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.151939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:47925)
0928 02:46:22.152060 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:22.152095 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260928 02:46:22.274505 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:22.274236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:33773)
0928 02:46:22.274353 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:22.274409 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260928 02:46:23.086767 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:23.086574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:33211)
0928 02:46:23.086690 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:23.086723 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 02:46:23.152760 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:23.152565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:53725)
0928 02:46:23.152685 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:23.152718 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260928 02:46:23.275115 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:23.274920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:33733)
0928 02:46:23.275044 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:23.275079 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 02:46:24.087426 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.087204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:55755)
0928 02:46:24.087322 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.087356 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 02:46:24.153380 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.153183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:41439)
0928 02:46:24.153302 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.153338 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 02:46:24.275703 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:24.275525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:38121)
0928 02:46:24.275635 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:24.275668 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260928 02:46:24.763347 10928 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:24.763440 10928 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:24.763617 10928 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
W20260928 02:46:24.764012 10521 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
W20260928 02:46:24.764103 10524 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:24.764168 10524 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6146a8c98cfc4846995b85524ab839b9; no voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966
I20260928 02:46:24.764259 10928 raft_consensus.cc:2726] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:46:25.088095 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.087876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:60745)
0928 02:46:25.088017 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.088050 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260928 02:46:25.153993 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.153789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:47183)
0928 02:46:25.153914 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.153948 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260928 02:46:25.276297 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:25.276091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:58367)
0928 02:46:25.276226 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:25.276260 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260928 02:46:26.088860 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.088636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:56027)
0928 02:46:26.088774 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.088813 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260928 02:46:26.154717 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.154508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:32983)
0928 02:46:26.154638 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.154676 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 02:46:26.276899 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:26.276724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:37533)
0928 02:46:26.276828 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:26.276863 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260928 02:46:27.089516 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:27.089307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:53255)
0928 02:46:27.089430 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:27.089467 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260928 02:46:27.155330 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:27.155134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:46833)
0928 02:46:27.155255 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:27.155289 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260928 02:46:27.277480 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:27.277293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:53313)
0928 02:46:27.277402 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:27.277436 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260928 02:46:28.090207 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:28.089996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:55725)
0928 02:46:28.090129 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:28.090161 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 02:46:28.155939 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:28.155758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:49839)
0928 02:46:28.155868 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:28.155900 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260928 02:46:28.278107 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:28.277900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:48919)
0928 02:46:28.278033 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:28.278068 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260928 02:46:28.786375 10941 raft_consensus.cc:482] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:28.786512 10941 raft_consensus.cc:504] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf4b8eab51db4474a6631d696d145966" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 42595 } } peers { permanent_uuid: "6146a8c98cfc4846995b85524ab839b9" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 39387 } } peers { permanent_uuid: "0bcb6f9f5704469e955955d03d52899c" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 46687 } }
I20260928 02:46:28.786720 10941 leader_election.cc:288] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595), 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687)
W20260928 02:46:28.787079 10524 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf4b8eab51db4474a6631d696d145966 (127.27.109.131:42595): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
W20260928 02:46:28.787170 10521 leader_election.cc:341] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb6f9f5704469e955955d03d52899c (127.27.109.130:46687): Not found: Tablet not found: 153b3057cca54ddb9f8397f3f6732c44
I20260928 02:46:28.787251 10521 leader_election.cc:302] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6146a8c98cfc4846995b85524ab839b9; no voters: 0bcb6f9f5704469e955955d03d52899c, cf4b8eab51db4474a6631d696d145966
I20260928 02:46:28.787328 10941 raft_consensus.cc:2726] T 153b3057cca54ddb9f8397f3f6732c44 P 6146a8c98cfc4846995b85524ab839b9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:46:29.090850 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.090650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:49083)
0928 02:46:29.090772 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.090805 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 02:46:29.156638 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.156428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:46683)
0928 02:46:29.156550 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.156583 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 02:46:29.278785 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:29.278581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:39955)
0928 02:46:29.278693 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:29.278728 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 02:46:30.091650 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:30.091411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:52449)
0928 02:46:30.091546 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:30.091586 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260928 02:46:30.157310 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:30.157115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:49353)
0928 02:46:30.157230 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:30.157267 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260928 02:46:30.279371 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:30.279164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:43425)
0928 02:46:30.279300 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:30.279331 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 02:46:31.092442 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.092187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:50023)
0928 02:46:31.092359 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.092396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260928 02:46:31.157986 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.157761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:55855)
0928 02:46:31.157903 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.157941 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 02:46:31.279991 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:31.279785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:34673)
0928 02:46:31.279912 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:31.279949 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260928 02:46:32.093189 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.092966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:58031)
0928 02:46:32.093096 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.093134 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260928 02:46:32.158629 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.158409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:32891)
0928 02:46:32.158557 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.158590 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260928 02:46:32.280679 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:32.280411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:55587)
0928 02:46:32.280595 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:32.280633 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260928 02:46:33.094007 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.093763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.129:47165)
0928 02:46:33.093926 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.093962 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260928 02:46:33.159368 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.159138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.130:37159)
0928 02:46:33.159292 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.159324 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260928 02:46:33.279495 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10476
I20260928 02:46:33.281371 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:33.281176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37383 (local address 127.27.109.131:33361)
0928 02:46:33.281299 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:33.281334 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:37383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260928 02:46:33.284524 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10639
I20260928 02:46:33.288655 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10772
2026-09-28T02:46:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18176 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:46:33 dist-test-slave-b5p5 krb5kdc[10959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:46:33 dist-test-slave-b5p5 krb5kdc[10959](info): set up 2 sockets
Sep 28 02:46:33 dist-test-slave-b5p5 krb5kdc[10959](info): commencing operation
krb5kdc: starting...
W20260928 02:46:34.322721 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:46:34.439132 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:34.517838 10972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:34.518079 10972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:34.518113 10972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:34.519241 10972 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:46:34.519313 10972 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:46:34.519335 10972 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:46:34.519352 10972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:34.519381 10972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:34.519410 10972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:34.519434 10972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:34.519464 10972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:34.520890 10972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:34.521165 10972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:34.521389 10972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:34.526557 10980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:34.526566 10977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.526594 10972 server_base.cc:1034] running on GCE node
W20260928 02:46:34.526569 10978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.527246 10972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:34.527516 10972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:34.528695 10972 hybrid_clock.cc:648] HybridClock initialized: now 1790563594528674 us; error 37 us; skew 500 ppm
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:34.531071 10972 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:34.531476 10972 webserver.cc:460] Webserver started at http://127.27.109.190:39647/ using document root <none> and password file <none>
I20260928 02:46:34.531706 10972 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:34.531774 10972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:34.531922 10972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:34.533046 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "d8581b0a242941269d27b7dafb0ad084"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "4f0caf3f7318932f8cf2fd7d0d995cdc"
server_key_iv: "3951a1c542bd9c3fb4bf5c119ee79731"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.533393 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "d8581b0a242941269d27b7dafb0ad084"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "4f0caf3f7318932f8cf2fd7d0d995cdc"
server_key_iv: "3951a1c542bd9c3fb4bf5c119ee79731"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.534731 10972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:46:34.535398 10987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.535563 10972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:34.535671 10972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "d8581b0a242941269d27b7dafb0ad084"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "4f0caf3f7318932f8cf2fd7d0d995cdc"
server_key_iv: "3951a1c542bd9c3fb4bf5c119ee79731"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.535758 10972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:34.572713 10972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:34.574659 10972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:34.574833 10972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:34.578505 10972 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39733
I20260928 02:46:34.578527 11039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39733 every 8 connection(s)
I20260928 02:46:34.578910 10972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:34.579741 11040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:34.581971 11040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap starting.
I20260928 02:46:34.582609 11040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:34.582909 11040 log.cc:824] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:34.583403 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 10972
I20260928 02:46:34.583504 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:46:34.583673 11040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: No bootstrap required, opened a new log
I20260928 02:46:34.583712 28086 external_mini_cluster.cc:1428] Setting key 652685155932b905a6d8d75727b376f6
I20260928 02:46:34.585533 11040 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:34.585690 11040 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:34.585737 11040 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Initialized, Role: FOLLOWER
I20260928 02:46:34.585884 11040 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:34.585978 11040 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:34.586021 11040 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:34.586067 11040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:34.586617 11040 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:34.586730 11040 leader_election.cc:302] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8581b0a242941269d27b7dafb0ad084; no voters:
I20260928 02:46:34.586922 11040 leader_election.cc:288] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:46:34.586946 11045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:34.587051 11045 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 LEADER]: Becoming Leader. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Running, Role: LEADER
I20260928 02:46:34.587190 11040 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:34.587190 11045 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:34.587718 11046 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8581b0a242941269d27b7dafb0ad084. Latest consensus state: current_term: 1 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:46:34.587718 11047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:46:34.587831 11046 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:34.587833 11047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:34.588153 11054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:34.588847 11054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:34.590608 11054 catalog_manager.cc:1347] Generated new cluster ID: a4b9c428dd6a401d90e2fe642f744e7e
I20260928 02:46:34.590667 11054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:34.595225 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.584198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.27.109.190:39733)
0928 02:46:34.584391 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.584400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.584432 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:46:34.584894 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.584900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.584946 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.585088 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:34.585715 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.587236 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.588433 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.589078 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.589216 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.591941 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:34.591945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:34.591946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:34.591958 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:34.593612 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.593988 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.593991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.593991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.594076 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.594282 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.594283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.594283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.594577 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:34.594717 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.594854 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.595083 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260928 02:46:34.604401 11054 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:46:34.605062 11054 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:34.617427 11054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Generated new TSK 0
I20260928 02:46:34.617583 11054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:46:34.653962 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:34.732769 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:34.732960 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:34.732982 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:34.734149 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:34.734192 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:34.734223 11066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:34.735711 11066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:34.735953 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:34.736171 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:34.741046 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:34.741077 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.741310 11066 server_base.cc:1034] running on GCE node
W20260928 02:46:34.741444 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.741645 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:34.741842 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:34.742975 11066 hybrid_clock.cc:648] HybridClock initialized: now 1790563594742967 us; error 31 us; skew 500 ppm
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:34.745347 11066 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:34.745771 11066 webserver.cc:460] Webserver started at http://127.27.109.129:44893/ using document root <none> and password file <none>
I20260928 02:46:34.746002 11066 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:34.746047 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:34.746202 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:34.747284 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "9ffd55603533499582ae2dc967960ff4"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "2c3af6316bd2aaa2a3bd0c0bfe1e9d36"
server_key_iv: "631d6651e0a892598ae27fc6c304a106"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.747649 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "9ffd55603533499582ae2dc967960ff4"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "2c3af6316bd2aaa2a3bd0c0bfe1e9d36"
server_key_iv: "631d6651e0a892598ae27fc6c304a106"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.748935 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:46:34.749609 11081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.749802 11066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:46:34.749889 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9ffd55603533499582ae2dc967960ff4"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "2c3af6316bd2aaa2a3bd0c0bfe1e9d36"
server_key_iv: "631d6651e0a892598ae27fc6c304a106"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.749963 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:34.760484 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:34.762960 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:34.763127 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:34.763362 11066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:34.763687 11066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:34.763742 11066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.763785 11066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:34.763813 11066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.769407 11066 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37005
I20260928 02:46:34.769479 11194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37005 every 8 connection(s)
I20260928 02:46:34.769793 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:34.778882 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.770555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:46199 (local address 127.27.109.190:39733)
0928 02:46:34.770615 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.770617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.770809 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:46:34.771285 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.771286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.771296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.771351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:34.772359 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.772868 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.773873 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.774074 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.774130 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.776117 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:34.776119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:34.776119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:34.776127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:34.777491 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.777901 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.777902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.777903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.777961 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.778213 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.778214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.778215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.778329 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:34.778404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.778702 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.778785 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:46:34.779076 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.770554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:46199)
0928 02:46:34.770708 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.770742 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.771198 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.771414 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.771417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.772006 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:34.772255 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.772260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.772949 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.772950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.773802 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.773804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.774145 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.774146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.774239 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.774711 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:34.774719 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:34.776020 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:34.777588 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:34.777596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:34.777598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:34.777825 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:34.778030 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:34.778030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:34.778031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:34.778140 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:34.778394 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:34.778400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:34.778571 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.778682 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.778845 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260928 02:46:34.779385 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11066
I20260928 02:46:34.779523 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:46:34.779644 11195 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:34.779723 28086 external_mini_cluster.cc:1428] Setting key 0610dc1b41f8808889972621d434b71c
I20260928 02:46:34.779786 11195 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:34.779953 11195 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
I20260928 02:46:34.780455 11004 ts_manager.cc:194] Registered new tserver with Master: 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:34.781068 11004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:46199
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:46:34.813138 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:34.893160 11200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:34.893345 11200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:34.893368 11200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:34.894513 11200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:34.894553 11200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:34.894582 11200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:34.896021 11200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:34.896253 11200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:34.896467 11200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:34.901219 11205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:34.901263 11206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.901276 11200 server_base.cc:1034] running on GCE node
W20260928 02:46:34.901232 11208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:34.901643 11200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:34.901831 11200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:34.902985 11200 hybrid_clock.cc:648] HybridClock initialized: now 1790563594902976 us; error 31 us; skew 500 ppm
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:34.905432 11200 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:34.905860 11200 webserver.cc:460] Webserver started at http://127.27.109.130:38415/ using document root <none> and password file <none>
I20260928 02:46:34.906071 11200 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:34.906137 11200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:34.906286 11200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:34.907398 11200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "28c881332fbf4d938677a2ac68e6b58f"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "860360655bece568a13801fe476b8d09"
server_key_iv: "77551dabae516d3dd9784824cfe82363"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.907822 11200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "28c881332fbf4d938677a2ac68e6b58f"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "860360655bece568a13801fe476b8d09"
server_key_iv: "77551dabae516d3dd9784824cfe82363"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.909053 11200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:34.909730 11215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.909904 11200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:34.909997 11200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "28c881332fbf4d938677a2ac68e6b58f"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "860360655bece568a13801fe476b8d09"
server_key_iv: "77551dabae516d3dd9784824cfe82363"
server_key_version: "encryptionkey@0"
I20260928 02:46:34.910079 11200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:34.936401 11200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:34.938788 11200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:34.938941 11200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:34.939162 11200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:34.939474 11200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:34.939527 11200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.939568 11200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:34.939596 11200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:34.945068 11200 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:33341
I20260928 02:46:34.945152 11328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:33341 every 8 connection(s)
I20260928 02:46:34.945453 11200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:34.947796 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11200
I20260928 02:46:34.947898 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:46:34.948081 28086 external_mini_cluster.cc:1428] Setting key ac294a4f71c6cf428b122bd46d41a723
Sep 28 02:46:34 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563594, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:34.954870 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.946130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:43035 (local address 127.27.109.190:39733)
0928 02:46:34.946205 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:34.946208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:34.946380 (+ 172us) server_negotiation.cc:407] Connection header received
0928 02:46:34.946935 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:34.946936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:34.946947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:34.947011 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:34.948031 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.948590 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.949561 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.949788 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.949873 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.952002 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:34.952004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:34.952004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:34.952012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:34.953432 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.953896 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.953897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.953897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.953957 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:34.954205 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:34.954206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:34.954206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:34.954317 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:34.954392 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:34.954697 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 02:46:34.954781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:46:34.955058 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:34.946111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:43035)
0928 02:46:34.946263 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:34.946295 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:34.946827 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:34.947061 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:34.947064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:34.947637 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:34.947926 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.947932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.948689 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.948690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.949471 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:34.949473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:34.949883 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:34.949885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:34.950008 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:34.950545 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:34.950552 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:34.951904 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:34.953533 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:34.953540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:34.953542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:34.953821 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:34.954026 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:34.954027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:34.954027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:34.954136 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:34.954384 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:34.954390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:34.954567 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:46:34.954674 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:46:34.954827 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260928 02:46:34.955508 11329 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:34.955595 11329 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:34.955782 11329 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:46:34.956133 11004 ts_manager.cc:194] Registered new tserver with Master: 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
I20260928 02:46:34.956521 11004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:43035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:46:34.972448 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:35.050284 11334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:35.050518 11334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:35.050576 11334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:35.051712 11334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:35.051753 11334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:35.051800 11334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:35.053293 11334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:35.053521 11334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:35.053730 11334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:35.058653 11339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:35.058653 11342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:35.058658 11340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:35.058673 11334 server_base.cc:1034] running on GCE node
I20260928 02:46:35.059092 11334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:35.059334 11334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:35.060497 11334 hybrid_clock.cc:648] HybridClock initialized: now 1790563595060481 us; error 41 us; skew 500 ppm
Sep 28 02:46:35 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563595, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:35.063113 11334 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:35.063514 11334 webserver.cc:460] Webserver started at http://127.27.109.131:33649/ using document root <none> and password file <none>
I20260928 02:46:35.063726 11334 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:35.063769 11334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:35.063915 11334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:46:35.064883 11334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "44fccf9a5cf141248ca524c315318bbe"
format_stamp: "Formatted at 2026-09-28 02:46:35 on dist-test-slave-b5p5"
server_key: "b864f54710b163a0375d563eed9f84e3"
server_key_iv: "cd4f62c77b6ddd7fbe5a038d30cf8613"
server_key_version: "encryptionkey@0"
I20260928 02:46:35.065228 11334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "44fccf9a5cf141248ca524c315318bbe"
format_stamp: "Formatted at 2026-09-28 02:46:35 on dist-test-slave-b5p5"
server_key: "b864f54710b163a0375d563eed9f84e3"
server_key_iv: "cd4f62c77b6ddd7fbe5a038d30cf8613"
server_key_version: "encryptionkey@0"
I20260928 02:46:35.066530 11334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:46:35.067260 11349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:35.067472 11334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:35.067574 11334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "44fccf9a5cf141248ca524c315318bbe"
format_stamp: "Formatted at 2026-09-28 02:46:35 on dist-test-slave-b5p5"
server_key: "b864f54710b163a0375d563eed9f84e3"
server_key_iv: "cd4f62c77b6ddd7fbe5a038d30cf8613"
server_key_version: "encryptionkey@0"
I20260928 02:46:35.067659 11334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:35.087729 11334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:35.090081 11334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:35.090248 11334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:35.090512 11334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:35.090821 11334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:35.090873 11334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:35.090906 11334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:35.090934 11334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:35.096891 11334 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39633
I20260928 02:46:35.096940 11462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39633 every 8 connection(s)
I20260928 02:46:35.097240 11334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:46:35 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563595, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:35.106187 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.097903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:40811 (local address 127.27.109.190:39733)
0928 02:46:35.097979 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.097982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.098208 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:46:35.098747 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.098749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.098758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.098812 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:35.099654 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.100157 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.101106 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.101302 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.101366 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.103488 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:35.103489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:35.103490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:35.103502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:35.104956 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:35.105369 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:35.105371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:35.105371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:35.105441 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:35.105675 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:35.105676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:35.105677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:35.105779 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:35.105837 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.106057 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.106122 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:46:35.106418 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.097896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:40811)
0928 02:46:35.098086 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.098122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.098642 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.098865 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.098869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.099399 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:35.099574 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.099578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.100236 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.100236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.101010 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.101012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.101379 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.101380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.101474 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.101973 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:35.101981 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:35.103377 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:35.105051 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:35.105056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:35.105058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:35.105275 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:35.105521 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:35.105522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:35.105522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:35.105616 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:35.105843 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:35.105846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:35.105951 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.106031 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.106194 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260928 02:46:35.106693 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11334
I20260928 02:46:35.106791 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:46:35.106983 28086 external_mini_cluster.cc:1428] Setting key 924edf6d3a9b498a1d777c14c7b5aec9
I20260928 02:46:35.106989 11463 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:35.107072 11463 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:35.107254 11463 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
I20260928 02:46:35.107738 11003 ts_manager.cc:194] Registered new tserver with Master: 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633)
I20260928 02:46:35.108152 11003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:40811
I20260928 02:46:35.118902 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:35.126068 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.120270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.27.109.190:39733)
0928 02:46:35.120344 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.120346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.120421 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:46:35.120466 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.120467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.120475 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.120528 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:35.121066 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.121532 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.122240 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.122425 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.122543 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.123723 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:35.123724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:35.123724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:35.123729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:35.124901 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:35.125214 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:35.125214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:35.125215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:35.125263 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:35.125467 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:35.125468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:35.125468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:35.125644 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:35.125724 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.125947 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.126008 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:46:35.128002 11003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:35.129578 11003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:35.136300 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.131986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33341 (local address 127.0.0.1:44648)
0928 02:46:35.132131 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.132166 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.132232 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.132741 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.132742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.132744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:35.132802 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.132804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.134088 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.134090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.135175 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.135177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.135989 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.135990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.136066 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.136079 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.136159 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.136209 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":46,"threads_started":1}
I20260928 02:46:35.136312 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.132168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.27.109.130:33341)
0928 02:46:35.132564 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.132568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.132576 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:46:35.132632 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.132635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.132650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.132738 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:35.132888 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.133989 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.135270 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.135907 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.135965 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.136001 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.136189 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.136230 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":91,"threads_started":1}
I20260928 02:46:35.136626 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.131986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.0.0.1:50204)
0928 02:46:35.132254 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.132266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.132317 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.132726 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.132727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.132729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:35.132803 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.132804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.134548 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.134549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.135554 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.135557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.136402 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.136403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.136483 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.136488 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.136535 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.136567 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":11,"thread_start_us":86,"threads_started":1}
I20260928 02:46:35.136860 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.132366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.27.109.131:39633)
0928 02:46:35.132527 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.132531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.132540 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:35.132575 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.132577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.132591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.132661 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:35.132887 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.133982 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.135638 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.136248 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.136313 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.136367 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.136733 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.136782 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260928 02:46:35.137099 11263 tablet_service.cc:1467] Processing CreateTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (DEFAULT_TABLE table=test-table [id=d71b851d4e004e45a3c9264d225a680f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:35.137478 11263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:35.137521 11397 tablet_service.cc:1467] Processing CreateTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (DEFAULT_TABLE table=test-table [id=d71b851d4e004e45a3c9264d225a680f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:35.137884 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:35.140492 11480 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Bootstrap starting.
I20260928 02:46:35.141106 11481 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Bootstrap starting.
I20260928 02:46:35.141165 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.132008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.0.0.1:47218)
0928 02:46:35.132393 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.132402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.132438 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.132645 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.132648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.132649 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:35.132877 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.132879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.133881 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.133882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.135175 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.135177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.140935 (+ 5758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.140936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.140982 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.140985 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.141058 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.141101 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":92,"threads_started":1}
I20260928 02:46:35.141196 11480 tablet_bootstrap.cc:654] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:35.141214 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.132263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.27.109.129:37005)
0928 02:46:35.132426 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.132432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.132443 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:46:35.132515 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.132517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.132535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.132622 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:35.132986 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.133773 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.136033 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.140852 (+ 4819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.140937 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.140988 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.141077 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.141123 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260928 02:46:35.141860 11480 log.cc:824] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:35.141954 11481 tablet_bootstrap.cc:654] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:35.141886 11129 tablet_service.cc:1467] Processing CreateTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (DEFAULT_TABLE table=test-table [id=d71b851d4e004e45a3c9264d225a680f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:35.142218 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:35.142506 11481 log.cc:824] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:35.142721 11480 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: No bootstrap required, opened a new log
I20260928 02:46:35.142802 11480 ts_tablet_manager.cc:1397] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:35.143294 11481 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: No bootstrap required, opened a new log
I20260928 02:46:35.143359 11481 ts_tablet_manager.cc:1397] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:35.144919 11484 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Bootstrap starting.
I20260928 02:46:35.145042 11480 raft_consensus.cc:348] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.145275 11480 raft_consensus.cc:374] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:35.145344 11480 raft_consensus.cc:729] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c881332fbf4d938677a2ac68e6b58f, State: Initialized, Role: FOLLOWER
I20260928 02:46:35.145273 11481 raft_consensus.cc:348] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.145464 11481 raft_consensus.cc:374] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:35.145485 11480 consensus_queue.cc:260] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.145537 11481 raft_consensus.cc:729] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fccf9a5cf141248ca524c315318bbe, State: Initialized, Role: FOLLOWER
I20260928 02:46:35.145596 11484 tablet_bootstrap.cc:654] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:35.145669 11481 consensus_queue.cc:260] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.145772 11480 ts_tablet_manager.cc:1428] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:46:35.145925 11329 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
I20260928 02:46:35.145934 11481 ts_tablet_manager.cc:1428] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:46:35.146199 11484 log.cc:824] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:35.146217 11463 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
I20260928 02:46:35.146934 11484 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: No bootstrap required, opened a new log
I20260928 02:46:35.147023 11484 ts_tablet_manager.cc:1397] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:46:35.148578 11484 raft_consensus.cc:348] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.148725 11484 raft_consensus.cc:374] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:35.148772 11484 raft_consensus.cc:729] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffd55603533499582ae2dc967960ff4, State: Initialized, Role: FOLLOWER
I20260928 02:46:35.148880 11484 consensus_queue.cc:260] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.149122 11484 ts_tablet_manager.cc:1428] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:46:35.149190 11195 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
W20260928 02:46:35.196014 11330 tablet.cc:2369] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:35.244110 11485 raft_consensus.cc:482] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:35.244285 11485 raft_consensus.cc:504] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.244674 11485 leader_election.cc:288] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:35.247921 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.244961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:46129 (local address 127.27.109.129:37005)
0928 02:46:35.245022 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.245024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.245058 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:35.245091 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.245092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.245102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.245163 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:35.245369 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.245920 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247177 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.247551 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247618 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.247648 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.247806 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.247847 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:46:35.247932 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.244980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:58103 (local address 127.27.109.131:39633)
0928 02:46:35.245037 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.245039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.245205 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:46:35.245279 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.245280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.245290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.245342 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:35.245510 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.245985 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247174 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.247556 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247665 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.247673 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.247836 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.247871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:46:35.247932 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.244880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.27.109.130:46129)
0928 02:46:35.244947 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.244958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.245003 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.245205 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.245206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.245208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:35.245283 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.245285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.246006 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.246007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.247082 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.247085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.247642 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.247728 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.247731 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.247787 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.247847 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260928 02:46:35.247944 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.244880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.27.109.130:58103)
0928 02:46:35.245126 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.245137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:35.245207 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:35.245366 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:35.245367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:35.245369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:35.245441 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.245442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.246068 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.246068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.247084 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:35.247087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.247678 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.247679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:35.247742 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.247746 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:46:35.247819 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:46:35.247854 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":30,"thread_start_us":130,"threads_started":1}
I20260928 02:46:35.248324 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdac72fa3a54d82b7dbf65c3aa5b0af" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffd55603533499582ae2dc967960ff4" is_pre_election: true
I20260928 02:46:35.248488 11149 raft_consensus.cc:2454] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 0.
I20260928 02:46:35.248564 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdac72fa3a54d82b7dbf65c3aa5b0af" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fccf9a5cf141248ca524c315318bbe" is_pre_election: true
I20260928 02:46:35.248733 11417 raft_consensus.cc:2454] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 0.
I20260928 02:46:35.248746 11216 leader_election.cc:302] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c881332fbf4d938677a2ac68e6b58f, 9ffd55603533499582ae2dc967960ff4; no voters:
I20260928 02:46:35.248889 11485 raft_consensus.cc:2781] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:35.248948 11485 raft_consensus.cc:482] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:35.248977 11485 raft_consensus.cc:3037] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:35.249739 11485 raft_consensus.cc:504] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.249890 11485 leader_election.cc:288] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 election: Requested vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:35.250041 11417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdac72fa3a54d82b7dbf65c3aa5b0af" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fccf9a5cf141248ca524c315318bbe"
I20260928 02:46:35.250097 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdac72fa3a54d82b7dbf65c3aa5b0af" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffd55603533499582ae2dc967960ff4"
I20260928 02:46:35.250119 11417 raft_consensus.cc:3037] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:35.250239 11149 raft_consensus.cc:3037] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:35.250924 11417 raft_consensus.cc:2454] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 1.
I20260928 02:46:35.251091 11218 leader_election.cc:302] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe; no voters:
I20260928 02:46:35.251160 11149 raft_consensus.cc:2454] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 1.
I20260928 02:46:35.251188 11485 raft_consensus.cc:2781] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:35.251339 11485 raft_consensus.cc:686] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 1 LEADER]: Becoming Leader. State: Replica: 28c881332fbf4d938677a2ac68e6b58f, State: Running, Role: LEADER
I20260928 02:46:35.251437 11485 consensus_queue.cc:237] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:35.252261 11004 catalog_manager.cc:5495] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130). New cstate: current_term: 1 leader_uuid: "28c881332fbf4d938677a2ac68e6b58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } health_report { overall_health: UNKNOWN } } }
W20260928 02:46:35.270512 11196 tablet.cc:2369] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:46:35.279814 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.277451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.27.109.130:33341)
0928 02:46:35.277495 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:35.277497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:35.277505 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:46:35.277573 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:35.277574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:35.277589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:35.277639 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:35.277752 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.278203 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.279004 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:35.279183 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:35.279281 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:35.279459 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:35.279562 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:35.279656 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:35.279677 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 02:46:35.279697 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:46:35.281548 11417 raft_consensus.cc:1261] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Refusing update from remote peer 28c881332fbf4d938677a2ac68e6b58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:35.281558 11149 raft_consensus.cc:1261] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Refusing update from remote peer 28c881332fbf4d938677a2ac68e6b58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:35.281908 11485 consensus_queue.cc:1035] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:35.282037 11490 consensus_queue.cc:1035] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:35.285225 11495 mvcc.cc:205] Tried to move back new op lower bound from 7334148486270631936 to 7334148486150352896. Current Snapshot: MvccSnapshot[applied={T|T < 7334148486270631936 or (T in {7334148486270631936})}]
I20260928 02:46:35.286602 11003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:35.286698 11003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:35.287446 11003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Sending DeleteTablet for 3 replicas of tablet ffdac72fa3a54d82b7dbf65c3aa5b0af
I20260928 02:46:35.287761 11263 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:35 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:44648
I20260928 02:46:35.287812 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11066
I20260928 02:46:35.287994 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:35 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:47218
I20260928 02:46:35.288631 11502 tablet_replica.cc:330] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: stopping tablet replica
I20260928 02:46:35.288738 11502 raft_consensus.cc:2229] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:35.288836 11502 raft_consensus.cc:2258] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:35.289287 11502 ts_tablet_manager.cc:1905] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:35.290856 11397 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:35 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:50204
I20260928 02:46:35.291085 11503 tablet_replica.cc:330] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: stopping tablet replica
I20260928 02:46:35.291214 11503 raft_consensus.cc:2229] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:35.291370 11503 raft_consensus.cc:2258] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:35.291507 11502 ts_tablet_manager.cc:1918] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:35.291620 11502 log.cc:1196] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/ffdac72fa3a54d82b7dbf65c3aa5b0af
I20260928 02:46:35.291923 11503 ts_tablet_manager.cc:1905] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:35.291956 11502 ts_tablet_manager.cc:1939] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 28c881332fbf4d938677a2ac68e6b58f: Deleting consensus metadata
I20260928 02:46:35.292385 10991 catalog_manager.cc:4841] TS 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (table test-table [id=d71b851d4e004e45a3c9264d225a680f]) successfully deleted
I20260928 02:46:35.293522 11503 ts_tablet_manager.cc:1918] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:35.293694 11503 log.cc:1196] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/ffdac72fa3a54d82b7dbf65c3aa5b0af
I20260928 02:46:35.294003 11503 ts_tablet_manager.cc:1939] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 44fccf9a5cf141248ca524c315318bbe: Deleting consensus metadata
I20260928 02:46:35.294386 10990 catalog_manager.cc:4841] TS 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (table test-table [id=d71b851d4e004e45a3c9264d225a680f]) successfully deleted
W20260928 02:46:35.294797 10988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:37005 (error 108)
I20260928 02:46:35.295075 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11200
I20260928 02:46:35.295869 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:35.295204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.0.0.1:47224)
0928 02:46:35.295806 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:35.295843 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575}
W20260928 02:46:35.295984 10988 catalog_manager.cc:4584] TS 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37005: connect: Connection refused (error 111)
I20260928 02:46:35.300472 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11334
I20260928 02:46:35.305015 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 10972
I20260928 02:46:55.310431 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--webserver_interface=127.27.109.190
--webserver_port=39647
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:46:55.389941 11505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:55.390180 11505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:55.390216 11505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:55.391335 11505 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:46:55.391395 11505 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:46:55.391417 11505 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:46:55.391438 11505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:46:55.391470 11505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:55.391498 11505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:55.391521 11505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:46:55.391552 11505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:46:55.392946 11505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:55.393209 11505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:55.393415 11505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:55.401484 11511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.401491 11512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.401535 11514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.401551 11505 server_base.cc:1034] running on GCE node
I20260928 02:46:55.401973 11505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:55.402285 11505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:55.403453 11505 hybrid_clock.cc:648] HybridClock initialized: now 1790563615403429 us; error 51 us; skew 500 ppm
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:55.409852 11505 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:46:55.410349 11505 webserver.cc:460] Webserver started at http://127.27.109.190:39647/ using document root <none> and password file <none>
I20260928 02:46:55.410642 11505 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:55.410735 11505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:55.412447 11505 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:46:55.413252 11521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.413403 11505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:55.413502 11505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "d8581b0a242941269d27b7dafb0ad084"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "4f0caf3f7318932f8cf2fd7d0d995cdc"
server_key_iv: "3951a1c542bd9c3fb4bf5c119ee79731"
server_key_version: "encryptionkey@0"
I20260928 02:46:55.413879 11505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:55.423905 11505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:55.425580 11505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:55.425720 11505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:55.429610 11505 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39733
I20260928 02:46:55.429643 11573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39733 every 8 connection(s)
I20260928 02:46:55.429994 11505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:46:55.431361 11574 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:46:55.431995 11574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap starting.
I20260928 02:46:55.435313 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11505
I20260928 02:46:55.435640 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:37005
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=44893
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:55.440913 11574 log.cc:824] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:55.442711 11574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:55.442993 11574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap complete.
I20260928 02:46:55.445147 11574 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:55.445355 11574 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Initialized, Role: FOLLOWER
I20260928 02:46:55.445509 11574 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:55.445566 11574 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:46:55.445634 11574 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:46:55.445688 11574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:46:55.446714 11574 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:55.446796 11574 leader_election.cc:302] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8581b0a242941269d27b7dafb0ad084; no voters:
I20260928 02:46:55.446988 11574 leader_election.cc:288] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:46:55.447021 11578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:46:55.447139 11578 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 LEADER]: Becoming Leader. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Running, Role: LEADER
I20260928 02:46:55.447245 11574 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:46:55.447290 11578 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:46:55.447562 11578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8581b0a242941269d27b7dafb0ad084. Latest consensus state: current_term: 2 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:46:55.447566 11579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:46:55.447711 11578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:55.447741 11579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:46:55.449349 11594 catalog_manager.cc:1259] Loaded cluster ID: a4b9c428dd6a401d90e2fe642f744e7e
I20260928 02:46:55.449388 11594 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: loading cluster ID for follower catalog manager: success
I20260928 02:46:55.450119 11594 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: acquiring CA information for follower catalog manager: success
I20260928 02:46:55.450548 11594 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:46:55.450618 11586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:46:55.450956 11586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:46:55.451017 11586 catalog_manager.cc:1259] Loaded cluster ID: a4b9c428dd6a401d90e2fe642f744e7e
I20260928 02:46:55.451038 11586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:46:55.451191 11586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:46:55.451254 11586 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Loaded TSK: 0
I20260928 02:46:55.464789 11586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Generated new TSK 1
I20260928 02:46:55.464876 11586 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:46:55.516069 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:55.516330 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:55.516373 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:55.517642 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:55.517699 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:55.517740 11576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:46:55.519527 11576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=44893
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:55.519763 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:55.520012 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:55.524904 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.525098 11576 server_base.cc:1034] running on GCE node
W20260928 02:46:55.525133 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.524989 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.525418 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:55.525687 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:55.526857 11576 hybrid_clock.cc:648] HybridClock initialized: now 1790563615526844 us; error 27 us; skew 500 ppm
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:55.529630 11576 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:46:55.530145 11576 webserver.cc:460] Webserver started at http://127.27.109.129:44893/ using document root <none> and password file <none>
I20260928 02:46:55.530350 11576 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:55.530426 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:55.531847 11576 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:55.532569 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.532742 11576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:55.532816 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9ffd55603533499582ae2dc967960ff4"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "2c3af6316bd2aaa2a3bd0c0bfe1e9d36"
server_key_iv: "631d6651e0a892598ae27fc6c304a106"
server_key_version: "encryptionkey@0"
I20260928 02:46:55.533183 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:55.567363 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:55.569597 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:55.569739 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:55.569975 11576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:55.570508 11618 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:46:55.571913 11576 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:46:55.571971 11576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:46:55.572027 11576 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:46:55.572649 11576 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:46:55.572698 11576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:46:55.572739 11618 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Bootstrap starting.
I20260928 02:46:55.580363 11576 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37005
I20260928 02:46:55.580441 11725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37005 every 8 connection(s)
I20260928 02:46:55.580840 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:46:55.584275 11618 log.cc:824] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:55.586292 11618 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:46:55.586621 11618 tablet_bootstrap.cc:492] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Bootstrap complete.
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:55.587733 11618 ts_tablet_manager.cc:1397] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260928 02:46:55.588793 11618 raft_consensus.cc:348] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.589110 11618 raft_consensus.cc:729] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffd55603533499582ae2dc967960ff4, State: Initialized, Role: FOLLOWER
I20260928 02:46:55.589270 11618 consensus_queue.cc:260] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.589562 11618 ts_tablet_manager.cc:1428] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 02:46:55.590812 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11576
I20260928 02:46:55.591077 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:33341
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=38415
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:46:55.591626 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.581673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:45567 (local address 127.27.109.190:39733)
0928 02:46:55.581854 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.581862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.581920 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:46:55.582519 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.582525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.582579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.582703 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.583324 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.584177 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.585219 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.585604 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.585734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.588173 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:55.588176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:55.588177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:55.588188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:55.589931 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.590317 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.590319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.590320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.590407 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.590706 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.590706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.590707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.590944 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:55.591017 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.591349 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.591471 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260928 02:46:55.591624 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.581560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:45567)
0928 02:46:55.581783 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.581832 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.582300 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.582752 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.582756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.583011 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.583190 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.583195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.584305 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.584307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.585129 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.585131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.585733 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.585735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.585873 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.586430 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.586438 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.588059 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:55.590020 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.590027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.590029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.590239 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.590492 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.590493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.590493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.590615 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.591012 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:55.591018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:55.591146 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.591268 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.591429 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
I20260928 02:46:55.592190 11726 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:55.592276 11726 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:55.592453 11726 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
I20260928 02:46:55.593176 11538 ts_manager.cc:194] Registered new tserver with Master: 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:55.594166 11538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:45567
I20260928 02:46:55.595043 11726 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.129@KRBTEST.COM
I20260928 02:46:55.601490 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.593756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.27.109.129:37005)
0928 02:46:55.593934 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.593938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.593981 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:46:55.594086 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.594087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.594097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.594153 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.594635 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.595569 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.596283 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.596564 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.596645 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.598804 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:55.598810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:55.598811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:55.598816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:55.600237 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.600702 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.600704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.600705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.600757 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.601005 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.601005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.601006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.601113 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:55.601169 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.601353 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.601415 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260928 02:46:55.601617 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.593717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.0.0.1:42978)
0928 02:46:55.593889 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.593923 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.594003 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.594171 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.594176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.594512 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.594565 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.594567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.595629 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.595630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.596188 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.596190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.596637 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.596638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.596713 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.597162 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.597166 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.598715 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:55.600317 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.600325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.600327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.600622 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.600821 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.600822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.600822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.600923 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.601174 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:55.601181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:55.601237 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.601335 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.601475 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260928 02:46:55.601981 11642 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdac72fa3a54d82b7dbf65c3aa5b0af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:35 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:42978
I20260928 02:46:55.602397 11735 tablet_replica.cc:330] stopping tablet replica
I20260928 02:46:55.602523 11735 raft_consensus.cc:2229] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:55.602587 11735 raft_consensus.cc:2258] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:55.602833 11735 ts_tablet_manager.cc:1905] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:55.604493 11735 ts_tablet_manager.cc:1918] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:55.604547 11735 log.cc:1196] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal/wals/ffdac72fa3a54d82b7dbf65c3aa5b0af
I20260928 02:46:55.604784 11735 ts_tablet_manager.cc:1939] T ffdac72fa3a54d82b7dbf65c3aa5b0af P 9ffd55603533499582ae2dc967960ff4: Deleting consensus metadata
I20260928 02:46:55.605211 11522 catalog_manager.cc:4841] TS 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005): tablet ffdac72fa3a54d82b7dbf65c3aa5b0af (table test-table [id=d71b851d4e004e45a3c9264d225a680f]) successfully deleted
W20260928 02:46:55.673790 11732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:55.674006 11732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:55.674042 11732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:55.675177 11732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:55.675239 11732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:55.675283 11732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:46:55.676746 11732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=38415
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:55.676983 11732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:55.677214 11732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:55.682060 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.682161 11732 server_base.cc:1034] running on GCE node
W20260928 02:46:55.682078 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.682060 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.682505 11732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:55.682758 11732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:55.683912 11732 hybrid_clock.cc:648] HybridClock initialized: now 1790563615683884 us; error 57 us; skew 500 ppm
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:55.686348 11732 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:46:55.686795 11732 webserver.cc:460] Webserver started at http://127.27.109.130:38415/ using document root <none> and password file <none>
I20260928 02:46:55.687028 11732 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:55.687103 11732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:55.688416 11732 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:46:55.688933 11750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.689090 11732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:46:55.689179 11732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "28c881332fbf4d938677a2ac68e6b58f"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "860360655bece568a13801fe476b8d09"
server_key_iv: "77551dabae516d3dd9784824cfe82363"
server_key_version: "encryptionkey@0"
I20260928 02:46:55.689507 11732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:55.705053 11732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:55.706904 11732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:55.707058 11732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:55.707264 11732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:55.707549 11732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:55.707602 11732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.707639 11732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:55.707671 11732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.713745 11732 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:33341
I20260928 02:46:55.713791 11863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:33341 every 8 connection(s)
I20260928 02:46:55.714115 11732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:46:55.717250 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11732
I20260928 02:46:55.717559 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:39633
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=33649
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:55.724290 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.714766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:60311)
0928 02:46:55.714944 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.714983 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.715499 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.715697 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.715700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.716229 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.716396 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.716400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.717234 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.717235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.718578 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.718581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.719023 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.719025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.719150 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.719687 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.719695 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.721141 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:55.723007 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.723037 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.723039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.723248 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.723457 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.723457 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.723458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.723552 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.723832 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:55.723836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:55.723935 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.724009 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.724145 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260928 02:46:55.724301 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.714797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:60311 (local address 127.27.109.190:39733)
0928 02:46:55.714859 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.714861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.715088 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:46:55.715580 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.715581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.715594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.715650 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.716474 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.717120 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.718657 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.718915 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.719013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.721237 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:55.721239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:55.721239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:55.721247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:55.722928 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.723323 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.723324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.723325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.723386 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.723614 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.723615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.723615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.723761 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:55.723831 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.724054 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.724163 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:46:55.724802 11864 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:55.724897 11864 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:55.725068 11864 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
I20260928 02:46:55.725466 11538 ts_manager.cc:194] Registered new tserver with Master: 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
I20260928 02:46:55.726001 11538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:60311
W20260928 02:46:55.797958 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:46:55.798142 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:46:55.798166 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:46:55.799296 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:46:55.799336 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:46:55.799387 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:46:55.800758 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=33649
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:46:55.800976 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:46:55.801200 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:46:55.805858 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.805866 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:46:55.805878 11875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:46:55.806161 11867 server_base.cc:1034] running on GCE node
I20260928 02:46:55.806325 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:46:55.806579 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:46:55.807729 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790563615807711 us; error 36 us; skew 500 ppm
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:46:55.810029 11867 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:46:55.810410 11867 webserver.cc:460] Webserver started at http://127.27.109.131:33649/ using document root <none> and password file <none>
I20260928 02:46:55.810631 11867 fs_manager.cc:359] Metadata directory not provided
I20260928 02:46:55.810684 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:46:55.811906 11867 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:46:55.812464 11882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.812608 11867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:46:55.812695 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "44fccf9a5cf141248ca524c315318bbe"
format_stamp: "Formatted at 2026-09-28 02:46:35 on dist-test-slave-b5p5"
server_key: "b864f54710b163a0375d563eed9f84e3"
server_key_iv: "cd4f62c77b6ddd7fbe5a038d30cf8613"
server_key_version: "encryptionkey@0"
I20260928 02:46:55.812958 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:46:55.827769 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:46:55.829424 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:46:55.829527 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:46:55.829689 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:46:55.829897 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:46:55.829926 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.829945 11867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:46:55.829967 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.836709 11867 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39633
I20260928 02:46:55.836771 11995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39633 every 8 connection(s)
I20260928 02:46:55.837092 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:46:55 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:46:55.842857 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 11867
I20260928 02:46:55.846354 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.837759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:37523 (local address 127.27.109.190:39733)
0928 02:46:55.837815 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.837817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.838004 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:46:55.838513 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.838515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.838528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.838579 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.839473 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.839966 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.840930 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.841139 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.841210 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.843340 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:55.843342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:55.843343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:55.843351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:55.844976 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.845411 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.845412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.845413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.845468 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.845729 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.845729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.845730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.845825 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:55.845877 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.846189 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.846263 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:46:55.846436 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.837746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:37523)
0928 02:46:55.837899 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.837931 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.838385 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.838651 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.838655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.839196 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:46:55.839370 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.839374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.840061 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.840062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.840834 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.840836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.841214 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.841215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.841336 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.841732 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:46:55.841739 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:46:55.843229 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:46:55.845055 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.845060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.845061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.845336 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.845523 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:46:55.845524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:46:55.845524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:46:55.845660 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:46:55.845886 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:46:55.845891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:46:55.846032 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.846144 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.846271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 02:46:55.846976 11996 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39733
I20260928 02:46:55.847071 11996 heartbeater.cc:464] Registering TS with master...
I20260928 02:46:55.847249 11996 heartbeater.cc:511] Master 127.27.109.190:39733 requested a full tablet report, sending...
I20260928 02:46:55.847674 11538 ts_manager.cc:194] Registered new tserver with Master: 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633)
I20260928 02:46:55.848148 11538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:37523
I20260928 02:46:55.850572 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.843242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.27.109.190:39733)
0928 02:46:55.843551 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.843553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.843560 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:55.843579 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.843579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.843590 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.843654 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:46:55.844376 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.845141 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.846167 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.846494 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.846574 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.848018 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:46:55.848019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:46:55.848019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:46:55.848026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:46:55.849468 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.849786 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.849787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.849788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.849841 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:46:55.850041 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:46:55.850042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:46:55.850042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:46:55.850151 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:46:55.850331 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.850366 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.850445 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":54,"threads_started":1}
I20260928 02:46:55.851282 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:46:55.853693 12000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.851639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.27.109.190:39733)
0928 02:46:55.851682 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.851684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.851751 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:46:55.851802 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.851802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.851826 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.851875 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:55.851961 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.852378 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.853089 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.853269 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.853315 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.853429 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:55.853504 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:55.853555 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.853637 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.853653 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:46:55.853922 11538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:46:55.855531 11538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:46:55.857748 11642 tablet_service.cc:1467] Processing CreateTablet for tablet 78b74876137147e79cf22d608b84d414 (DEFAULT_TABLE table=test-table [id=c2e01652c296418aa927bea424f4d293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:55.857937 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b74876137147e79cf22d608b84d414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:55.859483 11618 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Bootstrap starting.
I20260928 02:46:55.859825 11618 tablet_bootstrap.cc:654] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:55.861331 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.857242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33341 (local address 127.0.0.1:47774)
0928 02:46:55.857292 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.857302 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.857339 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.857776 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.857777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.857779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.857817 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.857818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.858889 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.858890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.860034 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.860038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.861111 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.861112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.861191 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.861213 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.861245 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.861289 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:46:55.861359 11618 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: No bootstrap required, opened a new log
I20260928 02:46:55.861398 11618 ts_tablet_manager.cc:1397] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:46:55.861626 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.857422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.27.109.130:33341)
0928 02:46:55.857612 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.857616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.857625 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:46:55.857658 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.857660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.857671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.857737 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.857891 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.858809 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.860187 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.861022 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.861450 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.861512 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.861540 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.861569 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
I20260928 02:46:55.861713 11618 raft_consensus.cc:348] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.861773 11618 raft_consensus.cc:374] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:55.861807 11618 raft_consensus.cc:729] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffd55603533499582ae2dc967960ff4, State: Initialized, Role: FOLLOWER
I20260928 02:46:55.861847 11618 consensus_queue.cc:260] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.861925 11618 ts_tablet_manager.cc:1428] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:46:55.862226 11798 tablet_service.cc:1467] Processing CreateTablet for tablet 78b74876137147e79cf22d608b84d414 (DEFAULT_TABLE table=test-table [id=c2e01652c296418aa927bea424f4d293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:55.862529 11798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b74876137147e79cf22d608b84d414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:55.863019 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.857242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.0.0.1:44890)
0928 02:46:55.859507 (+ 2265us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.859517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.859573 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.859763 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.859764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.859766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.859810 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.859812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.860997 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.860998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.861957 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.861960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.862866 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.862867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.862909 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.862911 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.862948 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.862975 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"mutex_wait_us":20,"thread_start_us":90,"threads_started":1}
I20260928 02:46:55.863183 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.857371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.27.109.131:39633)
0928 02:46:55.857536 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.857541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.859627 (+ 2086us) server_negotiation.cc:407] Connection header received
0928 02:46:55.859663 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.859665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.859679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.859748 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.859883 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.860926 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.862065 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.862753 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.862791 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.862820 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.862985 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.863028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260928 02:46:55.863658 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 78b74876137147e79cf22d608b84d414 (DEFAULT_TABLE table=test-table [id=c2e01652c296418aa927bea424f4d293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:46:55.863906 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b74876137147e79cf22d608b84d414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:46:55.865242 12006 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Bootstrap starting.
I20260928 02:46:55.865918 12006 tablet_bootstrap.cc:654] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:55.866369 12007 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Bootstrap starting.
I20260928 02:46:55.866534 12006 log.cc:824] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:55.867070 12007 tablet_bootstrap.cc:654] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Neither blocks nor log segments found. Creating new log.
I20260928 02:46:55.867245 12006 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: No bootstrap required, opened a new log
I20260928 02:46:55.867324 12006 ts_tablet_manager.cc:1397] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:55.867624 12007 log.cc:824] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Log is configured to *not* fsync() on all Append() calls
I20260928 02:46:55.868330 12007 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: No bootstrap required, opened a new log
I20260928 02:46:55.868402 12007 ts_tablet_manager.cc:1397] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:46:55.869331 12006 raft_consensus.cc:348] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.869505 12006 raft_consensus.cc:374] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:55.869553 12006 raft_consensus.cc:729] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c881332fbf4d938677a2ac68e6b58f, State: Initialized, Role: FOLLOWER
I20260928 02:46:55.869729 12006 consensus_queue.cc:260] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.870028 12006 ts_tablet_manager.cc:1428] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:46:55.870033 11864 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
I20260928 02:46:55.870373 12007 raft_consensus.cc:348] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.870582 12007 raft_consensus.cc:374] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:46:55.870643 12007 raft_consensus.cc:729] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fccf9a5cf141248ca524c315318bbe, State: Initialized, Role: FOLLOWER
I20260928 02:46:55.870774 12007 consensus_queue.cc:260] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.871059 11996 heartbeater.cc:503] Master 127.27.109.190:39733 was elected leader, sending a full tablet report...
I20260928 02:46:55.871265 12007 ts_tablet_manager.cc:1428] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:46:55.906169 12010 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:55.906286 12010 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.906651 12010 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:55.908594 11731 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:46:55.908713 11731 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.909015 11731 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
I20260928 02:46:55.910566 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.906843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.27.109.130:54779)
0928 02:46:55.906914 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.906925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.906963 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.907137 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.907138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.907140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.907218 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.907220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.908163 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.908164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.909495 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.909500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910313 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.910314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.910395 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.910398 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.910443 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.910495 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":93}
I20260928 02:46:55.910787 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.906985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:43449 (local address 127.27.109.131:39633)
0928 02:46:55.907024 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.907026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.907074 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:46:55.907146 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.907147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.907155 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.907202 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.907334 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.908069 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.909856 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910402 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.910444 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.910451 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.910691 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.910724 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260928 02:46:55.910818 11734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.906921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:54779 (local address 127.27.109.129:37005)
0928 02:46:55.906969 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.906971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.907005 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:46:55.907033 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.907034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.907058 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.907125 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.907296 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.908079 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.909564 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910154 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.910218 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.910281 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.910724 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.910761 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:46:55.911000 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.906858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.27.109.130:43449)
0928 02:46:55.907022 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.907030 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.907076 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.907213 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.907214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.907215 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.907270 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.907271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.908164 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.908164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.909764 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.909767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910539 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.910547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.910618 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.910621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.910874 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.910916 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":369,"thread_start_us":108,"threads_started":1}
I20260928 02:46:55.911023 11680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffd55603533499582ae2dc967960ff4" is_pre_election: true
I20260928 02:46:55.911114 11680 raft_consensus.cc:2454] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 0.
I20260928 02:46:55.911438 11751 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c881332fbf4d938677a2ac68e6b58f, 9ffd55603533499582ae2dc967960ff4; no voters:
I20260928 02:46:55.911420 11946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fccf9a5cf141248ca524c315318bbe" is_pre_election: true
I20260928 02:46:55.911551 11946 raft_consensus.cc:2454] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 0.
I20260928 02:46:55.911546 12010 raft_consensus.cc:2781] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:46:55.911588 12010 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:46:55.911628 12010 raft_consensus.cc:3037] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:55.912423 12010 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.912678 11946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fccf9a5cf141248ca524c315318bbe"
I20260928 02:46:55.912758 11946 raft_consensus.cc:3037] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:55.913123 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.909341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:33177 (local address 127.27.109.130:33341)
0928 02:46:55.909380 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.909382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.909756 (+ 374us) server_negotiation.cc:407] Connection header received
0928 02:46:55.909847 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.909848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.909856 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.909882 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.910099 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910777 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912182 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.912708 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912778 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.912784 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.913040 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.913075 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:46:55.913239 11728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.909145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.27.109.129:55713)
0928 02:46:55.909590 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.909599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.909638 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.909788 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.909789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.909791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.909846 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.909847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.911291 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.911292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.912294 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.912296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.912980 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.913050 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.913055 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.913123 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.913172 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429}
I20260928 02:46:55.913441 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.909154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33341 (local address 127.27.109.129:33177)
0928 02:46:55.909711 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.909719 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:46:55.909777 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:46:55.909934 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:46:55.909935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:46:55.909937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:46:55.909976 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.909978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910865 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.910866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.912089 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:46:55.912092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.912877 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:46:55.912940 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.912945 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:46:55.913031 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:46:55.913067 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":14,"thread_start_us":80,"threads_started":1}
I20260928 02:46:55.913507 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.909154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:55713 (local address 127.27.109.131:39633)
0928 02:46:55.909293 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.909294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.909651 (+ 357us) server_negotiation.cc:407] Connection header received
0928 02:46:55.909676 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.909677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.909685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.909900 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:46:55.909930 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.910831 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912368 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.912895 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.912968 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.912975 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.913394 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.913431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260928 02:46:55.913661 11950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "9ffd55603533499582ae2dc967960ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44fccf9a5cf141248ca524c315318bbe" is_pre_election: true
I20260928 02:46:55.913723 11946 raft_consensus.cc:2454] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 1.
I20260928 02:46:55.913738 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "9ffd55603533499582ae2dc967960ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c881332fbf4d938677a2ac68e6b58f" is_pre_election: true
I20260928 02:46:55.913831 11818 raft_consensus.cc:2379] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ffd55603533499582ae2dc967960ff4 in current term 1: Already voted for candidate 28c881332fbf4d938677a2ac68e6b58f in this term.
I20260928 02:46:55.913897 11753 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe; no voters:
I20260928 02:46:55.913985 12010 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [CANDIDATE]: Term 1 election: Requested vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005)
I20260928 02:46:55.913997 11680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b74876137147e79cf22d608b84d414" candidate_uuid: "28c881332fbf4d938677a2ac68e6b58f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffd55603533499582ae2dc967960ff4"
I20260928 02:46:55.914033 12010 raft_consensus.cc:2781] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:46:55.914057 11680 raft_consensus.cc:3037] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:46:55.914077 12010 raft_consensus.cc:686] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 LEADER]: Becoming Leader. State: Replica: 28c881332fbf4d938677a2ac68e6b58f, State: Running, Role: LEADER
I20260928 02:46:55.914127 11615 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:46:55.914148 12010 consensus_queue.cc:237] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:46:55.914942 11680 raft_consensus.cc:2454] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c881332fbf4d938677a2ac68e6b58f in term 1.
I20260928 02:46:55.915015 11731 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:46:55.914932 11538 catalog_manager.cc:5495] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f reported cstate change: term changed from 0 to 1, leader changed from <none> to 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130). New cstate: current_term: 1 leader_uuid: "28c881332fbf4d938677a2ac68e6b58f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } health_report { overall_health: UNKNOWN } } }
I20260928 02:46:55.921419 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.919433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.27.109.130:33341)
0928 02:46:55.919480 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:46:55.919486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:46:55.919493 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:46:55.919517 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:46:55.919518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:46:55.919530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:46:55.919592 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:46:55.919674 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.920135 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.920861 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:46:55.921028 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:46:55.921071 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:46:55.921182 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:46:55.921260 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:46:55.921313 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:46:55.921345 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:46:55.921358 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:46:55.922813 11680 raft_consensus.cc:1261] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Refusing update from remote peer 28c881332fbf4d938677a2ac68e6b58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:55.922840 11950 raft_consensus.cc:1261] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Refusing update from remote peer 28c881332fbf4d938677a2ac68e6b58f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:46:55.923090 12015 consensus_queue.cc:1035] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:55.923235 12010 consensus_queue.cc:1035] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:46:55.926055 12020 mvcc.cc:205] Tried to move back new op lower bound from 7334148570817613824 to 7334148570785034240. Current Snapshot: MvccSnapshot[applied={T|T < 7334148570817613824}]
I20260928 02:46:55.926883 11538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:55.926975 11538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:46:55.927625 11538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Sending DeleteTablet for 3 replicas of tablet 78b74876137147e79cf22d608b84d414
I20260928 02:46:55.927909 11919 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b74876137147e79cf22d608b84d414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:55 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:44890
I20260928 02:46:55.928035 11798 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b74876137147e79cf22d608b84d414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:46:55 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:47774
I20260928 02:46:55.928189 12026 tablet_replica.cc:330] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: stopping tablet replica
I20260928 02:46:55.928189 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11576
I20260928 02:46:55.928292 12026 raft_consensus.cc:2229] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:46:55.928386 12026 raft_consensus.cc:2258] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:55.929551 12026 ts_tablet_manager.cc:1905] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:46:55.931427 12026 ts_tablet_manager.cc:1918] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:46:55.931506 12026 log.cc:1196] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/78b74876137147e79cf22d608b84d414
I20260928 02:46:55.931988 12026 ts_tablet_manager.cc:1939] T 78b74876137147e79cf22d608b84d414 P 44fccf9a5cf141248ca524c315318bbe: Deleting consensus metadata
I20260928 02:46:55.932353 11524 catalog_manager.cc:4841] TS 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): tablet 78b74876137147e79cf22d608b84d414 (table test-table [id=c2e01652c296418aa927bea424f4d293]) successfully deleted
I20260928 02:46:55.934580 12027 tablet_replica.cc:330] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: stopping tablet replica
I20260928 02:46:55.934689 12027 raft_consensus.cc:2229] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:46:55.934777 12027 raft_consensus.cc:2258] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:46:55.935129 12027 ts_tablet_manager.cc:1905] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 02:46:55.935386 11522 connection.cc:535] client connection to 127.27.109.129:37005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:46:55.935463 11522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:46:55.935629 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11732
I20260928 02:46:55.935989 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.935878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37005 (local address 127.0.0.1:42990)
0928 02:46:55.935920 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.935952 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:46:55.936167 11522 catalog_manager.cc:4584] TS 9ffd55603533499582ae2dc967960ff4 (127.27.109.129:37005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78b74876137147e79cf22d608b84d414: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37005: connect: Connection refused (error 111)
I20260928 02:46:55.941102 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:46:55.940995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33341 (local address 127.0.0.1:47794)
0928 02:46:55.941053 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 02:46:55.941080 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.130:33341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:46:55.941252 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11867
I20260928 02:46:55.945811 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 11505
I20260928 02:47:15.951083 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--webserver_interface=127.27.109.190
--webserver_port=39647
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:16.030704 12029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:16.030884 12029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:16.030905 12029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:16.031960 12029 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:47:16.031993 12029 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:47:16.032007 12029 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:47:16.032018 12029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:16.032032 12029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:16.032045 12029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:16.032058 12029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:16.032073 12029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:16.033414 12029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:16.033586 12029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:16.033732 12029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:16.038604 12037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.038579 12035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:16.038620 12029 server_base.cc:1034] running on GCE node
W20260928 02:47:16.038591 12034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:16.039000 12029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:16.039219 12029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:16.040357 12029 hybrid_clock.cc:648] HybridClock initialized: now 1790563636040334 us; error 38 us; skew 500 ppm
Sep 28 02:47:16 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:16.046726 12029 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:16.047097 12029 webserver.cc:460] Webserver started at http://127.27.109.190:39647/ using document root <none> and password file <none>
I20260928 02:47:16.047331 12029 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:16.047376 12029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:16.048655 12029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:16.049319 12044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.049512 12029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:16.049603 12029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "d8581b0a242941269d27b7dafb0ad084"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "4f0caf3f7318932f8cf2fd7d0d995cdc"
server_key_iv: "3951a1c542bd9c3fb4bf5c119ee79731"
server_key_version: "encryptionkey@0"
I20260928 02:47:16.049911 12029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:16.060287 12029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:16.062453 12029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:16.062664 12029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:16.066310 12029 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39733
I20260928 02:47:16.066354 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39733 every 8 connection(s)
I20260928 02:47:16.066749 12029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:16.068210 12097 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:47:16.068845 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap starting.
I20260928 02:47:16.075692 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12029
I20260928 02:47:16.076045 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:37005
--local_ip_for_outbound_sockets=127.27.109.129
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=44893
--webserver_interface=127.27.109.129
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:47:16.077099 12097 log.cc:824] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:16.079403 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:47:16.079696 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Bootstrap complete.
I20260928 02:47:16.081820 12097 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:47:16.082052 12097 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Initialized, Role: FOLLOWER
I20260928 02:47:16.082250 12097 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:47:16.082345 12097 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:16.082396 12097 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:16.082450 12097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:47:16.083395 12097 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:47:16.083518 12097 leader_election.cc:302] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8581b0a242941269d27b7dafb0ad084; no voters:
I20260928 02:47:16.083684 12097 leader_election.cc:288] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:47:16.083739 12101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:47:16.083904 12101 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [term 3 LEADER]: Becoming Leader. State: Replica: d8581b0a242941269d27b7dafb0ad084, State: Running, Role: LEADER
I20260928 02:47:16.083926 12097 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:16.084048 12101 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } }
I20260928 02:47:16.084311 12102 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:47:16.084437 12102 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:16.084725 12109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:16.084785 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8581b0a242941269d27b7dafb0ad084. Latest consensus state: current_term: 3 leader_uuid: "d8581b0a242941269d27b7dafb0ad084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8581b0a242941269d27b7dafb0ad084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39733 } } }
I20260928 02:47:16.084874 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:16.085744 12109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:16.085906 12109 catalog_manager.cc:1259] Loaded cluster ID: a4b9c428dd6a401d90e2fe642f744e7e
I20260928 02:47:16.086088 12109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:16.086999 12109 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:16.087176 12109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Loaded TSK: 0
I20260928 02:47:16.087232 12109 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8581b0a242941269d27b7dafb0ad084: Loaded TSK: 1
F20260928 02:47:16.087440 12109 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1e661789ad google::LogMessage::Fail() at ??:0
@ 0x7f1e6617b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1e661784de google::LogMessage::Flush() at ??:0
@ 0x7f1e66179a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1e688e5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1e688e2d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1e6e0d21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1e6e0d248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1e6e0d2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1e6e098c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1e6e0a2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1e680460d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1e6803b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1e68e9f6db start_thread at ??:0
@ 0x7f1e64c7661f clone at ??:0
W20260928 02:47:16.176132 12099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:16.176306 12099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:16.176328 12099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:16.177438 12099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:16.177482 12099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:16.177508 12099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:16.178961 12099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:37005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=44893
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:16.179180 12099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:16.179436 12099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:16.184253 12138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.184266 12141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.184265 12139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:16.184525 12099 server_base.cc:1034] running on GCE node
I20260928 02:47:16.184703 12099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:16.184934 12099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:16.186102 12099 hybrid_clock.cc:648] HybridClock initialized: now 1790563636186073 us; error 47 us; skew 500 ppm
Sep 28 02:47:16 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:16.188539 12099 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:47:16.188980 12099 webserver.cc:460] Webserver started at http://127.27.109.129:44893/ using document root <none> and password file <none>
I20260928 02:47:16.189210 12099 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:16.189280 12099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:16.190658 12099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:47:16.191322 12148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.191509 12099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.191603 12099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "9ffd55603533499582ae2dc967960ff4"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "2c3af6316bd2aaa2a3bd0c0bfe1e9d36"
server_key_iv: "631d6651e0a892598ae27fc6c304a106"
server_key_version: "encryptionkey@0"
I20260928 02:47:16.191919 12099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:16.219071 12099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:16.221383 12099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:16.221551 12099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:16.221782 12099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:16.222211 12155 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:47:16.224231 12099 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:47:16.224291 12099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:47:16.224337 12099 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:47:16.224923 12099 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:47:16.224988 12099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:47:16.225033 12155 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Bootstrap starting.
I20260928 02:47:16.232089 12099 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37005
I20260928 02:47:16.232213 12262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37005 every 8 connection(s)
I20260928 02:47:16.232491 12099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:16.235893 12155 log.cc:824] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:16.238376 12155 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:47:16.238729 12155 tablet_bootstrap.cc:492] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Bootstrap complete.
I20260928 02:47:16.239939 12155 ts_tablet_manager.cc:1397] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260928 02:47:16.241245 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12099
I20260928 02:47:16.241513 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:33341
--local_ip_for_outbound_sockets=127.27.109.130
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=38415
--webserver_interface=127.27.109.130
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:47:16.241310 12155 raft_consensus.cc:348] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:16.241627 12155 raft_consensus.cc:729] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffd55603533499582ae2dc967960ff4, State: Initialized, Role: FOLLOWER
I20260928 02:47:16.241755 12155 consensus_queue.cc:260] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:16.242005 12155 ts_tablet_manager.cc:1428] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260928 02:47:16.273066 12265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:47:16.233136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:35073)
0928 02:47:16.233293 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.233335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:16.233789 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:16.272916 (+ 39127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260928 02:47:16.273311 12263 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:39733 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:39733: Client connection negotiation failed: client connection to 127.27.109.190:39733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:47:16.273634 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.273516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:54067)
0928 02:47:16.273570 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.273615 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:47:16.273923 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.273840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:53189)
0928 02:47:16.273883 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.273905 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:47:16.274029 12263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:47:16.323060 12268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:16.323269 12268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:16.323307 12268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:16.324424 12268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:16.324497 12268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:16.324542 12268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:16.325992 12268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:33341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=38415
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:16.326256 12268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:16.326511 12268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:16.331303 12276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.331354 12273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.331303 12274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:16.331377 12268 server_base.cc:1034] running on GCE node
I20260928 02:47:16.331673 12268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:16.331851 12268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:16.332971 12268 hybrid_clock.cc:648] HybridClock initialized: now 1790563636332958 us; error 27 us; skew 500 ppm
Sep 28 02:47:16 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:16.335337 12268 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:47:16.335789 12268 webserver.cc:460] Webserver started at http://127.27.109.130:38415/ using document root <none> and password file <none>
I20260928 02:47:16.336002 12268 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:16.336071 12268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:16.337294 12268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:47:16.337977 12283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.338140 12268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:16.338238 12268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "28c881332fbf4d938677a2ac68e6b58f"
format_stamp: "Formatted at 2026-09-28 02:46:34 on dist-test-slave-b5p5"
server_key: "860360655bece568a13801fe476b8d09"
server_key_iv: "77551dabae516d3dd9784824cfe82363"
server_key_version: "encryptionkey@0"
I20260928 02:47:16.338604 12268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:16.346205 12268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:16.347922 12268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:16.348073 12268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:16.348295 12268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:16.348685 12290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 02:47:16.349939 12290 ts_tablet_manager.cc:1852] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 02:47:16.350172 12290 ts_tablet_manager.cc:1905] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:16.352264 12290 ts_tablet_manager.cc:1918] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:16.352337 12290 log.cc:1196] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/78b74876137147e79cf22d608b84d414
I20260928 02:47:16.352628 12290 ts_tablet_manager.cc:1939] T 78b74876137147e79cf22d608b84d414 P 28c881332fbf4d938677a2ac68e6b58f: Deleting consensus metadata
I20260928 02:47:16.352835 12268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 02:47:16.352893 12268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 02:47:16.352953 12268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:16.352984 12268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.358974 12268 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:33341
I20260928 02:47:16.359030 12397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:33341 every 8 connection(s)
I20260928 02:47:16.359285 12268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:16.360181 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.359865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:40275)
0928 02:47:16.360000 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.360060 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260928 02:47:16.360411 12398 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:39733 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:39733: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
I20260928 02:47:16.360679 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.360598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:57745)
0928 02:47:16.360639 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.360664 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:47:16.360947 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.360871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:41537)
0928 02:47:16.360907 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.360927 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:47:16.361063 12398 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:47:16.366665 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12268
I20260928 02:47:16.366883 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:39633
--local_ip_for_outbound_sockets=127.27.109.131
--tserver_master_addrs=127.27.109.190:39733
--webserver_port=33649
--webserver_interface=127.27.109.131
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:16.445147 12401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:16.445328 12401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:16.445350 12401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:16.446688 12401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:16.446738 12401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:16.446769 12401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:16.448333 12401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=33649
--tserver_master_addrs=127.27.109.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:16.448561 12401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:16.448781 12401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:47:16.454037 12401 server_base.cc:1034] running on GCE node
W20260928 02:47:16.454005 12406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.454095 12407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:16.453996 12409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:16.454351 12401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:16.454614 12401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:16.455750 12401 hybrid_clock.cc:648] HybridClock initialized: now 1790563636455740 us; error 29 us; skew 500 ppm
Sep 28 02:47:16 dist-test-slave-b5p5 krb5kdc[10959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:16.458029 12401 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:47:16.458442 12401 webserver.cc:460] Webserver started at http://127.27.109.131:33649/ using document root <none> and password file <none>
I20260928 02:47:16.458707 12401 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:16.458765 12401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:16.460028 12401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:16.460608 12416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.460809 12401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:16.460922 12401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "44fccf9a5cf141248ca524c315318bbe"
format_stamp: "Formatted at 2026-09-28 02:46:35 on dist-test-slave-b5p5"
server_key: "b864f54710b163a0375d563eed9f84e3"
server_key_iv: "cd4f62c77b6ddd7fbe5a038d30cf8613"
server_key_version: "encryptionkey@0"
I20260928 02:47:16.461215 12401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:16.470363 12401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:16.472059 12401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:16.472203 12401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:16.472397 12401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:16.472671 12401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:16.472724 12401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.472756 12401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:16.472786 12401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:16.478724 12401 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39633
I20260928 02:47:16.478801 12529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39633 every 8 connection(s)
I20260928 02:47:16.479120 12401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:16.480037 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.479717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:41215)
0928 02:47:16.479853 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.479911 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
W20260928 02:47:16.480413 12530 heartbeater.cc:644] Failed to heartbeat to 127.27.109.190:39733 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.190:39733: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
I20260928 02:47:16.480670 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.480586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:40451)
0928 02:47:16.480628 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.480653 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:47:16.480919 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:16.480838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:53049)
0928 02:47:16.480874 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:16.480899 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:47:16.481022 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12401
W20260928 02:47:16.481032 12530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:47:17.274616 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.274321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:51007)
0928 02:47:17.274514 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:17.274552 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260928 02:47:17.361568 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.361369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:42877)
0928 02:47:17.361504 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:17.361537 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":53,"threads_started":1}
I20260928 02:47:17.481519 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.481332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:48271)
0928 02:47:17.481441 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:17.481474 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260928 02:47:17.495491 12537 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:17.495630 12537 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:17.495925 12537 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
Sep 28 02:47:17 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.131@KRBTEST.COM
Sep 28 02:47:17 dist-test-slave-b5p5 krb5kdc[10959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563636, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.130@KRBTEST.COM
I20260928 02:47:17.506587 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.496192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:58355 (local address 127.27.109.130:33341)
0928 02:47:17.496325 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:17.496333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:17.496353 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:47:17.496919 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:17.496922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:17.497013 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:17.497147 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:17.497792 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.498687 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.499683 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.500052 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.500189 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:17.503067 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:17.503071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:17.503071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:17.503080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:17.504915 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505352 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:17.505355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:17.505355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:17.505451 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505649 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:17.505650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:17.505650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:17.505927 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:17.506022 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:17.506334 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:47:17.506461 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
I20260928 02:47:17.506596 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.496158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:50251 (local address 127.27.109.131:39633)
0928 02:47:17.496324 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:17.496333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:17.496362 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:47:17.496829 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:17.496832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:17.496886 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:17.497021 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:17.497799 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.498715 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.499657 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.500052 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.500190 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:17.502618 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:17.502622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:17.502622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:17.502631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:17.504685 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505115 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:17.505117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:17.505118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:17.505199 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505578 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:17.505579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:17.505579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:17.505893 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:17.505986 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:17.506334 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:47:17.506502 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":53,"threads_started":1}
I20260928 02:47:17.506583 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.496092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39633 (local address 127.27.109.129:50251)
0928 02:47:17.496207 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:17.496215 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:17.496252 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:17.497098 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:17.497101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:17.497499 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:17.497677 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:17.497682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.498821 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.498822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:17.499566 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:17.499568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.500170 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.500171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:17.500266 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:17.500833 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:17.500842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:17.502523 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:17.504764 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:17.504770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:17.504771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:17.505020 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:17.505384 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505384 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:17.505385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:17.505498 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:17.505972 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:17.505994 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:17.506147 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:47:17.506263 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:47:17.506441 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 02:47:17.506649 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:17.496068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:33341 (local address 127.27.109.129:58355)
0928 02:47:17.496119 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:17.496130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:17.496180 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:17.497126 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:17.497127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:17.497500 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:17.497676 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:17.497682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.498773 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.498775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:17.499567 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:17.499570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:17.500146 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:17.500147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:17.500266 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:17.500887 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:17.500891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:17.502975 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:17.504991 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:17.504992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:17.504992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:17.505182 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:17.505519 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:17.505519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:17.505520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:17.505582 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:17.506011 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:17.506011 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:17.506147 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:47:17.506263 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:47:17.506529 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":186}
W20260928 02:47:17.507361 12151 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
W20260928 02:47:17.508001 12152 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
I20260928 02:47:17.508064 12152 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:47:17.508159 12537 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:47:18.275204 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:18.275011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:38083)
0928 02:47:18.275126 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:18.275159 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260928 02:47:18.362149 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:18.361964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:37821)
0928 02:47:18.362087 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:18.362119 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260928 02:47:18.482100 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:18.481917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:52337)
0928 02:47:18.482035 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:18.482067 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 02:47:19.275873 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:19.275678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:40919)
0928 02:47:19.275795 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:19.275827 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 02:47:19.362780 12545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:19.362600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:60667)
0928 02:47:19.362718 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:19.362747 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260928 02:47:19.453006 12546 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:19.453110 12546 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:19.453307 12546 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
W20260928 02:47:19.453758 12152 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
W20260928 02:47:19.453828 12151 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
I20260928 02:47:19.453871 12151 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:47:19.453954 12546 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:47:19.482698 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:19.482521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:48625)
0928 02:47:19.482633 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:19.482664 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 02:47:20.276505 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:20.276294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:56327)
0928 02:47:20.276428 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:20.276461 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 02:47:20.363312 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:20.363138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:42311)
0928 02:47:20.363244 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:20.363277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260928 02:47:20.483268 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:20.483076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:51873)
0928 02:47:20.483203 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:20.483235 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 02:47:21.261200 12551 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:21.261292 12551 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:21.261488 12551 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
W20260928 02:47:21.261884 12151 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
W20260928 02:47:21.261971 12152 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
I20260928 02:47:21.262038 12152 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:47:21.262115 12551 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:47:21.277151 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:21.276935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:57843)
0928 02:47:21.277085 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:21.277113 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260928 02:47:21.363878 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:21.363686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:36067)
0928 02:47:21.363808 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:21.363842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 02:47:21.483865 12554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:21.483672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:53109)
0928 02:47:21.483796 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:21.483828 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 02:47:22.277868 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:22.277572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:48341)
0928 02:47:22.277787 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:22.277822 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":58,"threads_started":1}
I20260928 02:47:22.364507 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:22.364279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:53467)
0928 02:47:22.364434 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:22.364469 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260928 02:47:22.484550 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:22.484367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:38961)
0928 02:47:22.484476 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:22.484517 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260928 02:47:23.278627 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:23.278392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:42599)
0928 02:47:23.278547 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:23.278581 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":58,"threads_started":1}
I20260928 02:47:23.365090 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:23.364900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:51791)
0928 02:47:23.365018 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:23.365050 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 02:47:23.485119 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:23.484934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:53295)
0928 02:47:23.485052 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:23.485086 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260928 02:47:24.279241 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:24.279030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:58301)
0928 02:47:24.279163 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:24.279196 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 02:47:24.365694 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:24.365514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:51555)
0928 02:47:24.365626 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:24.365658 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260928 02:47:24.485747 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:24.485556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:34387)
0928 02:47:24.485681 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:24.485710 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 02:47:25.279907 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:25.279694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:42227)
0928 02:47:25.279829 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:25.279862 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260928 02:47:25.366262 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:25.366075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:42281)
0928 02:47:25.366195 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:25.366227 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 02:47:25.486341 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:25.486160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:34829)
0928 02:47:25.486272 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:25.486302 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 02:47:26.250679 12567 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:26.250770 12567 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:26.250983 12567 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
W20260928 02:47:26.251425 12152 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
W20260928 02:47:26.251590 12151 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
I20260928 02:47:26.251660 12151 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:47:26.251756 12567 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:47:26.280580 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:26.280391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:43345)
0928 02:47:26.280495 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:26.280526 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
I20260928 02:47:26.366843 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:26.366668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:57539)
0928 02:47:26.366778 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:26.366808 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 02:47:26.486968 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:26.486775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:58243)
0928 02:47:26.486886 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:26.486918 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 02:47:27.281153 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:27.280972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:50779)
0928 02:47:27.281079 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:27.281109 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260928 02:47:27.367416 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:27.367241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:33179)
0928 02:47:27.367352 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:27.367383 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 02:47:27.487718 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:27.487467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:40265)
0928 02:47:27.487601 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:27.487638 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 02:47:28.281850 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:28.281632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:47171)
0928 02:47:28.281768 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:28.281803 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260928 02:47:28.368057 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:28.367869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:60797)
0928 02:47:28.367984 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:28.368019 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260928 02:47:28.488368 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:28.488170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:43429)
0928 02:47:28.488297 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:28.488330 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 02:47:29.282511 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:29.282264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:53825)
0928 02:47:29.282412 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:29.282445 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260928 02:47:29.368743 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:29.368547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:37231)
0928 02:47:29.368675 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:29.368708 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 02:47:29.489042 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:29.488811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:41567)
0928 02:47:29.488958 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:29.488992 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 02:47:29.707202 12580 raft_consensus.cc:482] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:29.707285 12580 raft_consensus.cc:504] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44fccf9a5cf141248ca524c315318bbe" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39633 } } peers { permanent_uuid: "28c881332fbf4d938677a2ac68e6b58f" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 33341 } } peers { permanent_uuid: "9ffd55603533499582ae2dc967960ff4" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37005 } }
I20260928 02:47:29.707481 12580 leader_election.cc:288] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633), 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341)
W20260928 02:47:29.707842 12151 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 44fccf9a5cf141248ca524c315318bbe (127.27.109.131:39633): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
W20260928 02:47:29.707896 12152 leader_election.cc:341] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c881332fbf4d938677a2ac68e6b58f (127.27.109.130:33341): Not found: Tablet not found: 78b74876137147e79cf22d608b84d414
I20260928 02:47:29.707928 12152 leader_election.cc:302] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffd55603533499582ae2dc967960ff4; no voters: 28c881332fbf4d938677a2ac68e6b58f, 44fccf9a5cf141248ca524c315318bbe
I20260928 02:47:29.708006 12580 raft_consensus.cc:2726] T 78b74876137147e79cf22d608b84d414 P 9ffd55603533499582ae2dc967960ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:47:30.283125 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:30.282934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:46557)
0928 02:47:30.283047 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:30.283081 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260928 02:47:30.369287 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:30.369107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:42587)
0928 02:47:30.369222 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:30.369250 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260928 02:47:30.489670 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:30.489470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:56151)
0928 02:47:30.489595 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:30.489627 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260928 02:47:31.284058 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:31.283709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.129:47399)
0928 02:47:31.283972 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:31.284012 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":195,"threads_started":1}
I20260928 02:47:31.370121 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:31.369868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.130:52613)
0928 02:47:31.370027 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:31.370073 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 02:47:31.490444 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:31.490197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39733 (local address 127.27.109.131:54723)
0928 02:47:31.490361 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:31.490405 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.190:39733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260928 02:47:31.492479 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12099
I20260928 02:47:31.497068 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12268
I20260928 02:47:31.501322 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12401
2026-09-28T02:47:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58803 ms)
[----------] 30 tests from SecurityITest (252959 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:32 dist-test-slave-b5p5 krb5kdc[12589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:32 dist-test-slave-b5p5 krb5kdc[12589](info): set up 2 sockets
Sep 28 02:47:32 dist-test-slave-b5p5 krb5kdc[12589](info): commencing operation
krb5kdc: starting...
W20260928 02:47:33.126163 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:33.237505 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:34571
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:33.317013 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:33.317242 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:33.317278 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:33.318405 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:33.318470 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:33.318552 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:33.318585 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:33.318611 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:33.320032 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:33.320288 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:33.320493 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:33.325484 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.325496 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.325552 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:33.325628 12602 server_base.cc:1034] running on GCE node
I20260928 02:47:33.325858 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:33.326131 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:33.327294 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790563653327281 us; error 44 us; skew 500 ppm
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:33.334435 12602 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:33.334923 12602 webserver.cc:460] Webserver started at http://127.27.109.190:35583/ using document root <none> and password file <none>
I20260928 02:47:33.335143 12602 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:33.335187 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:33.335330 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:33.336350 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "ff3853f66608457c9a2f0da79a3319bf"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "08aa6fcf9d45b050410e2c19ae500a74"
server_key_iv: "121e35bc4fbf632427f2d4a7bc9d1810"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.336702 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "ff3853f66608457c9a2f0da79a3319bf"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "08aa6fcf9d45b050410e2c19ae500a74"
server_key_iv: "121e35bc4fbf632427f2d4a7bc9d1810"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.337993 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:33.338814 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.338986 12602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:47:33.339088 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "ff3853f66608457c9a2f0da79a3319bf"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "08aa6fcf9d45b050410e2c19ae500a74"
server_key_iv: "121e35bc4fbf632427f2d4a7bc9d1810"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.339171 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:33.349488 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:33.351775 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:33.351931 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:33.355948 12602 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:34571
I20260928 02:47:33.355990 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:34571 every 8 connection(s)
I20260928 02:47:33.356359 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:33.357165 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:33.359392 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: Bootstrap starting.
I20260928 02:47:33.359949 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:33.360231 12670 log.cc:824] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:33.360914 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: No bootstrap required, opened a new log
I20260928 02:47:33.361855 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12602
I20260928 02:47:33.361950 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:33.362145 28086 external_mini_cluster.cc:1428] Setting key 228045e5b76f9a7a6b240633847a205e
I20260928 02:47:33.363081 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } }
I20260928 02:47:33.363256 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:33.363300 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3853f66608457c9a2f0da79a3319bf, State: Initialized, Role: FOLLOWER
I20260928 02:47:33.363435 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } }
I20260928 02:47:33.363526 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:33.363564 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:33.363615 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:33.364559 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } }
I20260928 02:47:33.364643 12670 leader_election.cc:302] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff3853f66608457c9a2f0da79a3319bf; no voters:
I20260928 02:47:33.364809 12670 leader_election.cc:288] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:33.364883 12675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:33.365010 12675 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [term 1 LEADER]: Becoming Leader. State: Replica: ff3853f66608457c9a2f0da79a3319bf, State: Running, Role: LEADER
I20260928 02:47:33.365072 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:33.365141 12675 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } }
I20260928 02:47:33.365581 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff3853f66608457c9a2f0da79a3319bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } } }
I20260928 02:47:33.365654 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:33.365859 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff3853f66608457c9a2f0da79a3319bf. Latest consensus state: current_term: 1 leader_uuid: "ff3853f66608457c9a2f0da79a3319bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3853f66608457c9a2f0da79a3319bf" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34571 } } }
I20260928 02:47:33.365955 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:33.366490 12682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:33.367204 12682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:33.368798 12682 catalog_manager.cc:1347] Generated new cluster ID: 5a3146038d8d4e368ad54395514ceeae
I20260928 02:47:33.368849 12682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:33.372890 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.362669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.27.109.190:34571)
0928 02:47:33.362842 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.362851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.362887 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:47:33.363042 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.363048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.363086 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.363188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:33.363995 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.365124 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.366253 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.366925 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.367051 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.369626 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:33.369630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:33.369630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:33.369641 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:33.371278 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.371649 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.371652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.371652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.371763 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.371970 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.371971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.371971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.372209 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:33.372372 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.372452 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.372586 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260928 02:47:33.378974 12682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:33.379761 12682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:33.409866 12682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: Generated new TSK 0
I20260928 02:47:33.410045 12682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:47:33.438455 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:33.537258 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:33.537443 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:33.537465 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:33.538656 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:33.538695 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:33.538721 12696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:33.540127 12696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:33.540338 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:33.540555 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:33.545430 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.545482 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.545431 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:33.545907 12696 server_base.cc:1034] running on GCE node
I20260928 02:47:33.546119 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:33.546310 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:33.547451 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790563653547430 us; error 35 us; skew 500 ppm
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:33.549891 12696 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:47:33.550302 12696 webserver.cc:460] Webserver started at http://127.27.109.129:46507/ using document root <none> and password file <none>
I20260928 02:47:33.550588 12696 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:33.550634 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:33.550786 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:33.551791 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "30d70064b25e49fa8f7e9d1ef38fa74f"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "6c347fa0abbcb3d4da887746e35728d9"
server_key_iv: "22d3456f2dccbf6bd15c77091822d5a3"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.552136 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "30d70064b25e49fa8f7e9d1ef38fa74f"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "6c347fa0abbcb3d4da887746e35728d9"
server_key_iv: "22d3456f2dccbf6bd15c77091822d5a3"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.553391 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:33.554107 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.554286 12696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:33.554383 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "30d70064b25e49fa8f7e9d1ef38fa74f"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "6c347fa0abbcb3d4da887746e35728d9"
server_key_iv: "22d3456f2dccbf6bd15c77091822d5a3"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.554469 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:33.593483 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:33.595566 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:33.595741 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:33.595976 12696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:33.596283 12696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:33.596338 12696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.596370 12696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:33.596397 12696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.601826 12696 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:38513
I20260928 02:47:33.601900 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:38513 every 8 connection(s)
I20260928 02:47:33.602196 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:33.603621 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12696
I20260928 02:47:33.603720 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:33.603922 28086 external_mini_cluster.cc:1428] Setting key 461e558a819699fef0a25d6cc97d02f3
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
I20260928 02:47:33.612505 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.603012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:45093 (local address 127.27.109.190:34571)
0928 02:47:33.603091 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.603093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.603275 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:47:33.603918 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.603920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.603931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.604002 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:33.605269 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.605744 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.606819 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.607143 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.607225 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.609339 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:33.609341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:33.609341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:33.609349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:33.610972 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.611341 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.611342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.611342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.611406 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.611782 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.611784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.611784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.611923 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:33.612014 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.612312 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.612407 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:47:33.612746 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.602980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:34571 (local address 127.27.109.129:45093)
0928 02:47:33.603143 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.603176 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.603805 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.604090 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.604094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.604909 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:33.605174 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.605181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.605844 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.605845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.606722 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.606725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.607222 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.607224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.607328 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.607862 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:33.607870 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:33.609244 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:33.611050 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.611056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.611057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.611265 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.611505 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.611506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.611506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.611665 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.612003 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:33.612010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:33.612164 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.612289 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.612454 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260928 02:47:33.613268 12825 heartbeater.cc:347] Connected to a master server at 127.27.109.190:34571
I20260928 02:47:33.613381 12825 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:33.613628 12825 heartbeater.cc:511] Master 127.27.109.190:34571 requested a full tablet report, sending...
I20260928 02:47:33.614259 12634 ts_manager.cc:194] Registered new tserver with Master: 30d70064b25e49fa8f7e9d1ef38fa74f (127.27.109.129:38513)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
I20260928 02:47:33.615079 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:45093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:47:33.628871 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:33.707051 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:33.707234 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:33.707257 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:33.708305 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:33.708338 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:33.708362 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:33.709692 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:33.709861 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:33.710022 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:33.714823 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.714826 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:33.714887 12830 server_base.cc:1034] running on GCE node
W20260928 02:47:33.714843 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:33.715201 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:33.715394 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:33.716579 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790563653716562 us; error 27 us; skew 500 ppm
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:33.719072 12830 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:47:33.719441 12830 webserver.cc:460] Webserver started at http://127.27.109.130:34775/ using document root <none> and password file <none>
I20260928 02:47:33.719628 12830 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:33.719668 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:33.719753 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:33.720690 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "eba1f7fae0c3474cbb92aca1530bbcdd"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "df13b28d5432135fbdab2475059142a9"
server_key_iv: "c69f841db09003394b209afde35fa257"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.720991 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "eba1f7fae0c3474cbb92aca1530bbcdd"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "df13b28d5432135fbdab2475059142a9"
server_key_iv: "c69f841db09003394b209afde35fa257"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.722259 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:47:33.723006 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.723152 12830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.723234 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "eba1f7fae0c3474cbb92aca1530bbcdd"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "df13b28d5432135fbdab2475059142a9"
server_key_iv: "c69f841db09003394b209afde35fa257"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.723285 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:33.736846 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:33.739022 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:33.739125 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:33.739292 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:33.739509 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:33.739538 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.739558 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:33.739570 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.745196 12830 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:34287
I20260928 02:47:33.745270 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:34287 every 8 connection(s)
I20260928 02:47:33.745561 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:33.753090 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12830
I20260928 02:47:33.753226 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:33.753412 28086 external_mini_cluster.cc:1428] Setting key f53998a77e18397597810e5f2fbb6883
I20260928 02:47:33.754664 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.746311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:44239 (local address 127.27.109.190:34571)
0928 02:47:33.746380 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.746382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.746605 (+ 223us) server_negotiation.cc:407] Connection header received
0928 02:47:33.747087 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.747088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.747098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.747153 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:33.748025 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.748506 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.749500 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.749676 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.749728 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.751763 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:33.751764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:33.751765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:33.751770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:33.753131 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.753571 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.753572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.753573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.753634 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.753940 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.753941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.753942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.754095 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:33.754162 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.754540 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.754603 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:47:33.754937 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.746282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:34571 (local address 127.27.109.130:44239)
0928 02:47:33.746481 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.746520 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.747006 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.747223 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.747226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.747760 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:33.747951 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.747956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.748591 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.748592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.749427 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.749430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.749748 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.749748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.749844 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.750240 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:33.750248 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:33.751682 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:33.753241 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.753247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.753249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.753510 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.753769 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.753769 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.753770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.753867 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.754177 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:33.754184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:33.754358 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.754495 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.754664 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260928 02:47:33.755383 12959 heartbeater.cc:347] Connected to a master server at 127.27.109.190:34571
I20260928 02:47:33.755477 12959 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:33.755676 12959 heartbeater.cc:511] Master 127.27.109.190:34571 requested a full tablet report, sending...
I20260928 02:47:33.756088 12634 ts_manager.cc:194] Registered new tserver with Master: eba1f7fae0c3474cbb92aca1530bbcdd (127.27.109.130:34287)
I20260928 02:47:33.756589 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:44239
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:47:33.776814 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:42983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:33.855386 12964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:33.855574 12964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:33.855597 12964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:33.856729 12964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:33.856769 12964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:33.856796 12964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:33.858307 12964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:42983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:34571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:33.858560 12964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:33.858778 12964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:33.863462 12969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.863463 12972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:33.863513 12970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:33.863572 12964 server_base.cc:1034] running on GCE node
I20260928 02:47:33.863843 12964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:33.864017 12964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:33.865180 12964 hybrid_clock.cc:648] HybridClock initialized: now 1790563653865163 us; error 31 us; skew 500 ppm
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:33.867857 12964 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:47:33.868256 12964 webserver.cc:460] Webserver started at http://127.27.109.131:37127/ using document root <none> and password file <none>
I20260928 02:47:33.868469 12964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:33.868534 12964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:33.868692 12964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:33.869659 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "a69c55915e8f495c8b1b45357306b48e"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "b546457b4e49c39d530904f925ab2343"
server_key_iv: "6b6dcc879346b13abaf3194f8ba29ca4"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.870007 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "a69c55915e8f495c8b1b45357306b48e"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "b546457b4e49c39d530904f925ab2343"
server_key_iv: "6b6dcc879346b13abaf3194f8ba29ca4"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.871407 12964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:33.872112 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.872277 12964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:33.872370 12964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "a69c55915e8f495c8b1b45357306b48e"
format_stamp: "Formatted at 2026-09-28 02:47:33 on dist-test-slave-b5p5"
server_key: "b546457b4e49c39d530904f925ab2343"
server_key_iv: "6b6dcc879346b13abaf3194f8ba29ca4"
server_key_version: "encryptionkey@0"
I20260928 02:47:33.872452 12964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:33.879441 12964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:33.881353 12964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:33.881500 12964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:33.881711 12964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:33.881978 12964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:33.882030 12964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.882068 12964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:33.882093 12964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:33.887945 12964 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:37829
I20260928 02:47:33.888031 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:37829 every 8 connection(s)
I20260928 02:47:33.888314 12964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:33.890658 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 12964
I20260928 02:47:33.890770 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:33.890937 28086 external_mini_cluster.cc:1428] Setting key 9f6c6f516463e9b779232ed30f810969
Sep 28 02:47:33 dist-test-slave-b5p5 krb5kdc[12589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563653, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:33.896832 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.888940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:38075 (local address 127.27.109.190:34571)
0928 02:47:33.888991 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.888993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.889199 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:47:33.889667 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.889668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.889678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.889734 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:33.890646 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.891405 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.892449 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.892629 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.892681 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.894677 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:33.894678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:33.894678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:33.894684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:33.895794 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.896129 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.896130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.896130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.896175 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.896369 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.896370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.896370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.896461 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:33.896509 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.896719 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.896772 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:47:33.897035 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.888920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:34571 (local address 127.27.109.131:38075)
0928 02:47:33.889059 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.889090 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.889577 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.889786 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.889789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.890327 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:33.890561 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.890566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.891496 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.891497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.892381 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.892383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.892702 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.892703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.892796 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.893292 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:33.893300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:33.894599 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:33.895861 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.895866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.895868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.896066 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.896232 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:33.896233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:33.896233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:33.896318 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:33.896515 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:33.896519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:33.896615 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.896695 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.896801 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260928 02:47:33.897473 13093 heartbeater.cc:347] Connected to a master server at 127.27.109.190:34571
I20260928 02:47:33.897567 13093 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:33.897725 13093 heartbeater.cc:511] Master 127.27.109.190:34571 requested a full tablet report, sending...
I20260928 02:47:33.898092 12634 ts_manager.cc:194] Registered new tserver with Master: a69c55915e8f495c8b1b45357306b48e (127.27.109.131:37829)
I20260928 02:47:33.898412 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:38075
I20260928 02:47:33.902509 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:33.909337 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.903723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.27.109.190:34571)
0928 02:47:33.903794 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.903796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.903899 (+ 103us) server_negotiation.cc:407] Connection header received
0928 02:47:33.903973 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.903974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.903982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.904033 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:33.904556 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.904997 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.905819 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.906004 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.906073 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.907158 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:33.907160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:33.907160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:33.907167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:33.908203 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.908543 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.908544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.908545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.908613 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:33.908827 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:33.908828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:33.908828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:33.908965 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:33.909070 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.909166 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.909244 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:47:33.910808 12634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:33.912236 12634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:33.918301 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:38513 (local address 127.0.0.1:34576)
0928 02:47:33.914433 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.914488 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.914551 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.914854 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.914857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.914859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:33.914924 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.914926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.915974 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.915975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.916945 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.916948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.917989 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.917990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.918064 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918077 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.918152 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.918209 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":43,"threads_started":1}
I20260928 02:47:33.918309 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.27.109.129:38513)
0928 02:47:33.914669 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.914673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.914684 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:33.914731 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.914733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.914747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.914837 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:33.914981 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.915885 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.917174 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.917906 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.917976 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918023 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.918176 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.918223 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260928 02:47:33.918732 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37829 (local address 127.0.0.1:60752)
0928 02:47:33.914577 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.914588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.914640 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.915061 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.915062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.915064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:33.915117 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.915119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.916559 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.916560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.917593 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.917596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.918482 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.918484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.918547 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.918608 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.918649 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
I20260928 02:47:33.918742 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.27.109.131:37829)
0928 02:47:33.914885 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.914888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.914897 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:47:33.914934 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.914936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.914946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.915035 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:33.915382 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.916452 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.917681 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.918393 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.918484 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918534 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.918621 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.918656 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":65,"threads_started":1}
I20260928 02:47:33.918637 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:34287 (local address 127.0.0.1:34530)
0928 02:47:33.915162 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:33.915237 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:33.915305 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:33.915486 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:33.915487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:33.915489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:33.915524 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.915526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.916716 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.916717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.917559 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:33.917562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.918386 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.918388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:33.918464 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918468 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:33.918532 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:47:33.918571 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":87,"threads_started":1}
I20260928 02:47:33.918844 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:33.914661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.27.109.130:34287)
0928 02:47:33.914829 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:33.914834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:33.915308 (+ 474us) server_negotiation.cc:407] Connection header received
0928 02:47:33.915379 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:33.915381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:33.915396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:33.915474 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:33.915586 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.916605 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.917643 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:33.918238 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:33.918301 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:33.918350 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:33.918711 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:47:33.918757 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260928 02:47:33.919265 12759 tablet_service.cc:1467] Processing CreateTablet for tablet 64dada955ebb41878944f8d376b1f609 (DEFAULT_TABLE table=test-table [id=92d779324db749f3ab01b535420b4831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:33.919338 13027 tablet_service.cc:1467] Processing CreateTablet for tablet 64dada955ebb41878944f8d376b1f609 (DEFAULT_TABLE table=test-table [id=92d779324db749f3ab01b535420b4831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:33.919643 12759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dada955ebb41878944f8d376b1f609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:33.919665 13027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dada955ebb41878944f8d376b1f609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:33.922019 13111 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: Bootstrap starting.
I20260928 02:47:33.922109 13110 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Bootstrap starting.
I20260928 02:47:33.922712 13111 tablet_bootstrap.cc:654] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:33.922806 13110 tablet_bootstrap.cc:654] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:33.922860 12893 tablet_service.cc:1467] Processing CreateTablet for tablet 64dada955ebb41878944f8d376b1f609 (DEFAULT_TABLE table=test-table [id=92d779324db749f3ab01b535420b4831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:33.923192 12893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64dada955ebb41878944f8d376b1f609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:33.923266 13111 log.cc:824] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:33.923377 13110 log.cc:824] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:33.924055 13111 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: No bootstrap required, opened a new log
I20260928 02:47:33.924082 13110 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: No bootstrap required, opened a new log
I20260928 02:47:33.924130 13111 ts_tablet_manager.cc:1397] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:33.924130 13110 ts_tablet_manager.cc:1397] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:47:33.926079 13114 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Bootstrap starting.
I20260928 02:47:33.926049 13111 raft_consensus.cc:348] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.926049 13110 raft_consensus.cc:348] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.926225 13111 raft_consensus.cc:374] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:33.926224 13110 raft_consensus.cc:374] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:33.926271 13111 raft_consensus.cc:729] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d70064b25e49fa8f7e9d1ef38fa74f, State: Initialized, Role: FOLLOWER
I20260928 02:47:33.926271 13110 raft_consensus.cc:729] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69c55915e8f495c8b1b45357306b48e, State: Initialized, Role: FOLLOWER
I20260928 02:47:33.926378 13111 consensus_queue.cc:260] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.926391 13110 consensus_queue.cc:260] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.926664 13110 ts_tablet_manager.cc:1428] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 02:47:33.926664 13111 ts_tablet_manager.cc:1428] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:47:33.926750 13114 tablet_bootstrap.cc:654] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:33.926803 13093 heartbeater.cc:503] Master 127.27.109.190:34571 was elected leader, sending a full tablet report...
I20260928 02:47:33.926836 12825 heartbeater.cc:503] Master 127.27.109.190:34571 was elected leader, sending a full tablet report...
I20260928 02:47:33.927297 13114 log.cc:824] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:33.928074 13114 tablet_bootstrap.cc:492] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: No bootstrap required, opened a new log
I20260928 02:47:33.928148 13114 ts_tablet_manager.cc:1397] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:47:33.929436 13114 raft_consensus.cc:348] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.929557 13114 raft_consensus.cc:374] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:33.929579 13114 raft_consensus.cc:729] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba1f7fae0c3474cbb92aca1530bbcdd, State: Initialized, Role: FOLLOWER
I20260928 02:47:33.929718 13114 consensus_queue.cc:260] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:33.929991 13114 ts_tablet_manager.cc:1428] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:47:33.930004 12959 heartbeater.cc:503] Master 127.27.109.190:34571 was elected leader, sending a full tablet report...
W20260928 02:47:33.996115 12960 tablet.cc:2369] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:34.063714 13116 raft_consensus.cc:482] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:34.063822 13116 raft_consensus.cc:504] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:34.064172 13116 leader_election.cc:288] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eba1f7fae0c3474cbb92aca1530bbcdd (127.27.109.130:34287), a69c55915e8f495c8b1b45357306b48e (127.27.109.131:37829)
I20260928 02:47:34.067104 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.064433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:43109 (local address 127.27.109.130:34287)
0928 02:47:34.064477 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.064479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.064486 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:47:34.064517 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.064517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.064523 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.064588 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.064771 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.065254 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066301 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.066749 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066842 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.066851 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.067003 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.067040 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:47:34.067104 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.064303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:34287 (local address 127.27.109.129:43109)
0928 02:47:34.064359 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.064371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.064416 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.064591 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.064592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.064594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.064670 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.064672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.065335 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.065337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.066211 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.066213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.066840 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.066919 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.066922 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.066988 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.067028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":28}
I20260928 02:47:34.067178 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.064410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:37967 (local address 127.27.109.131:37829)
0928 02:47:34.064455 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.064457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.064554 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:47:34.064633 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.064635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.064650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.064710 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.064914 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.065380 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066395 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.066823 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066907 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.066915 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.067075 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.067110 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:47:34.067216 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.064345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:37829 (local address 127.27.109.129:37967)
0928 02:47:34.064477 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.064486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.064568 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.064761 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.064762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.064763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.064840 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.064842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.065463 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.065464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.066305 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.066307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.066904 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.066905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.066979 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.066983 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.067065 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.067103 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260928 02:47:34.067446 12913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dada955ebb41878944f8d376b1f609" candidate_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" is_pre_election: true
I20260928 02:47:34.067489 13047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dada955ebb41878944f8d376b1f609" candidate_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69c55915e8f495c8b1b45357306b48e" is_pre_election: true
I20260928 02:47:34.067600 12913 raft_consensus.cc:2454] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d70064b25e49fa8f7e9d1ef38fa74f in term 0.
I20260928 02:47:34.067610 13047 raft_consensus.cc:2454] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d70064b25e49fa8f7e9d1ef38fa74f in term 0.
I20260928 02:47:34.067960 12714 leader_election.cc:302] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d70064b25e49fa8f7e9d1ef38fa74f, eba1f7fae0c3474cbb92aca1530bbcdd; no voters:
I20260928 02:47:34.068133 13116 raft_consensus.cc:2781] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:34.068171 13116 raft_consensus.cc:482] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:34.068186 13116 raft_consensus.cc:3037] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.068868 13116 raft_consensus.cc:504] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:34.069023 13116 leader_election.cc:288] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [CANDIDATE]: Term 1 election: Requested vote from peers eba1f7fae0c3474cbb92aca1530bbcdd (127.27.109.130:34287), a69c55915e8f495c8b1b45357306b48e (127.27.109.131:37829)
I20260928 02:47:34.069164 12913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dada955ebb41878944f8d376b1f609" candidate_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd"
I20260928 02:47:34.069238 12913 raft_consensus.cc:3037] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.069229 13047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64dada955ebb41878944f8d376b1f609" candidate_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69c55915e8f495c8b1b45357306b48e"
I20260928 02:47:34.069336 13047 raft_consensus.cc:3037] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.069953 12913 raft_consensus.cc:2454] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d70064b25e49fa8f7e9d1ef38fa74f in term 1.
I20260928 02:47:34.070116 13047 raft_consensus.cc:2454] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d70064b25e49fa8f7e9d1ef38fa74f in term 1.
I20260928 02:47:34.070158 12714 leader_election.cc:302] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d70064b25e49fa8f7e9d1ef38fa74f, eba1f7fae0c3474cbb92aca1530bbcdd; no voters:
I20260928 02:47:34.070268 13116 raft_consensus.cc:2781] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:34.070417 13116 raft_consensus.cc:686] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 1 LEADER]: Becoming Leader. State: Replica: 30d70064b25e49fa8f7e9d1ef38fa74f, State: Running, Role: LEADER
I20260928 02:47:34.070555 13116 consensus_queue.cc:237] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } }
I20260928 02:47:34.071367 12633 catalog_manager.cc:5495] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f reported cstate change: term changed from 0 to 1, leader changed from <none> to 30d70064b25e49fa8f7e9d1ef38fa74f (127.27.109.129). New cstate: current_term: 1 leader_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d70064b25e49fa8f7e9d1ef38fa74f" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:34.098170 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.095738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.27.109.129:38513)
0928 02:47:34.095808 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.095810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.095819 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:47:34.095859 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.095860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.095882 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.095963 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:34.096109 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.096610 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.097386 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.097584 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.097660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.097828 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:34.097931 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:34.098025 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.098044 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.098063 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:47:34.100039 13047 raft_consensus.cc:1261] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 1 FOLLOWER]: Refusing update from remote peer 30d70064b25e49fa8f7e9d1ef38fa74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:34.100061 12913 raft_consensus.cc:1261] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 1 FOLLOWER]: Refusing update from remote peer 30d70064b25e49fa8f7e9d1ef38fa74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:34.100360 13120 consensus_queue.cc:1035] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69c55915e8f495c8b1b45357306b48e" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:34.100492 13116 consensus_queue.cc:1035] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba1f7fae0c3474cbb92aca1530bbcdd" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:34.103016 13124 mvcc.cc:205] Tried to move back new op lower bound from 7334148727190929408 to 7334148727073742848. Current Snapshot: MvccSnapshot[applied={T|T < 7334148727190929408}]
I20260928 02:47:34.103227 13125 mvcc.cc:205] Tried to move back new op lower bound from 7334148727190929408 to 7334148727073742848. Current Snapshot: MvccSnapshot[applied={T|T < 7334148727190929408}]
I20260928 02:47:34.104830 12633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:34.104934 12633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:34.105671 12633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff3853f66608457c9a2f0da79a3319bf: Sending DeleteTablet for 3 replicas of tablet 64dada955ebb41878944f8d376b1f609
I20260928 02:47:34.105986 13027 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dada955ebb41878944f8d376b1f609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:60752
I20260928 02:47:34.106035 12893 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dada955ebb41878944f8d376b1f609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:34530
I20260928 02:47:34.106266 13133 tablet_replica.cc:330] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: stopping tablet replica
I20260928 02:47:34.106276 12759 tablet_service.cc:1514] Processing DeleteTablet for tablet 64dada955ebb41878944f8d376b1f609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:34576
I20260928 02:47:34.106359 13133 raft_consensus.cc:2229] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:34.106451 13133 raft_consensus.cc:2258] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.106544 13134 tablet_replica.cc:330] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: stopping tablet replica
I20260928 02:47:34.106642 13134 raft_consensus.cc:2229] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:34.106704 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12696
I20260928 02:47:34.106739 13134 raft_consensus.cc:2258] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.106449 13135 tablet_replica.cc:330] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f: stopping tablet replica
I20260928 02:47:34.106894 13135 raft_consensus.cc:2229] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:47:34.106917 13133 ts_tablet_manager.cc:1905] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:34.106992 13135 raft_consensus.cc:2258] T 64dada955ebb41878944f8d376b1f609 P 30d70064b25e49fa8f7e9d1ef38fa74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.107482 13134 ts_tablet_manager.cc:1905] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:34.108875 13134 ts_tablet_manager.cc:1918] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:34.108924 13134 log.cc:1196] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/64dada955ebb41878944f8d376b1f609
I20260928 02:47:34.109122 13134 ts_tablet_manager.cc:1939] T 64dada955ebb41878944f8d376b1f609 P eba1f7fae0c3474cbb92aca1530bbcdd: Deleting consensus metadata
I20260928 02:47:34.109165 13133 ts_tablet_manager.cc:1918] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:34.109238 13133 log.cc:1196] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/64dada955ebb41878944f8d376b1f609
I20260928 02:47:34.109434 12620 catalog_manager.cc:4841] TS eba1f7fae0c3474cbb92aca1530bbcdd (127.27.109.130:34287): tablet 64dada955ebb41878944f8d376b1f609 (table test-table [id=92d779324db749f3ab01b535420b4831]) successfully deleted
I20260928 02:47:34.109495 13133 ts_tablet_manager.cc:1939] T 64dada955ebb41878944f8d376b1f609 P a69c55915e8f495c8b1b45357306b48e: Deleting consensus metadata
I20260928 02:47:34.110325 12621 catalog_manager.cc:4841] TS a69c55915e8f495c8b1b45357306b48e (127.27.109.131:37829): tablet 64dada955ebb41878944f8d376b1f609 (table test-table [id=92d779324db749f3ab01b535420b4831]) successfully deleted
W20260928 02:47:34.113386 12620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:38513 (error 108)
I20260928 02:47:34.113723 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12830
I20260928 02:47:34.113920 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.113797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:38513 (local address 127.0.0.1:34582)
0928 02:47:34.113849 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.113885 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:38513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:47:34.114018 12620 catalog_manager.cc:4584] TS 30d70064b25e49fa8f7e9d1ef38fa74f (127.27.109.129:38513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64dada955ebb41878944f8d376b1f609: Network error: Client connection negotiation failed: client connection to 127.27.109.129:38513: connect: Connection refused (error 111)
I20260928 02:47:34.118299 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12964
I20260928 02:47:34.122928 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 12602
2026-09-28T02:47:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:34Z Disabled control of system clock
I20260928 02:47:34.149883 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39103
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:34.227852 13140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:34.228056 13140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:34.228078 13140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:34.229211 13140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:34.229251 13140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:34.229267 13140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:34.229303 13140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:34.229336 13140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:34.230633 13140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:34.230870 13140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:34.231141 13140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:34.235816 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.235864 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.236099 13140 server_base.cc:1034] running on GCE node
W20260928 02:47:34.235853 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.236398 13140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:34.236670 13140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:34.237835 13140 hybrid_clock.cc:648] HybridClock initialized: now 1790563654237808 us; error 39 us; skew 500 ppm
I20260928 02:47:34.239130 13140 webserver.cc:460] Webserver started at http://127.27.109.190:41511/ using document root <none> and password file <none>
I20260928 02:47:34.239364 13140 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:34.239411 13140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:34.239560 13140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:34.240589 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "5a489f10e628494aad90f61a7be87af1"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7e409ac23217b21850f0a51f68076aea"
server_key_iv: "5f61900621d03cc2499b40ee14e54465"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.240952 13140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "5a489f10e628494aad90f61a7be87af1"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7e409ac23217b21850f0a51f68076aea"
server_key_iv: "5f61900621d03cc2499b40ee14e54465"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.242234 13140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:34.243076 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.243244 13140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.243358 13140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "5a489f10e628494aad90f61a7be87af1"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7e409ac23217b21850f0a51f68076aea"
server_key_iv: "5f61900621d03cc2499b40ee14e54465"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.243429 13140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:34.263063 13140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:34.263356 13140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:34.263504 13140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:34.267381 13140 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39103
I20260928 02:47:34.267529 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39103 every 8 connection(s)
I20260928 02:47:34.267778 13140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:34.268805 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:34.271061 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: Bootstrap starting.
I20260928 02:47:34.271642 13207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:34.271950 13207 log.cc:824] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:34.272648 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: No bootstrap required, opened a new log
I20260928 02:47:34.273746 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13140
I20260928 02:47:34.273836 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:34.274118 28086 external_mini_cluster.cc:1428] Setting key 546ab0e8183d98327ada8f35422d40c0
I20260928 02:47:34.274372 13207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } }
I20260928 02:47:34.274648 13207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:34.274739 13207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a489f10e628494aad90f61a7be87af1, State: Initialized, Role: FOLLOWER
I20260928 02:47:34.274911 13207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } }
I20260928 02:47:34.275003 13207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:34.275033 13207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:34.275069 13207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.275782 13207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } }
I20260928 02:47:34.275892 13207 leader_election.cc:302] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a489f10e628494aad90f61a7be87af1; no voters:
I20260928 02:47:34.276033 13207 leader_election.cc:288] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:34.276108 13212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:34.276275 13207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:34.276281 13212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a489f10e628494aad90f61a7be87af1, State: Running, Role: LEADER
I20260928 02:47:34.276427 13212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } }
I20260928 02:47:34.276774 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a489f10e628494aad90f61a7be87af1. Latest consensus state: current_term: 1 leader_uuid: "5a489f10e628494aad90f61a7be87af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } } }
I20260928 02:47:34.277001 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:34.276932 13213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a489f10e628494aad90f61a7be87af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a489f10e628494aad90f61a7be87af1" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39103 } } }
I20260928 02:47:34.277415 13213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:34.277544 13221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:34.277958 13221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:34.279798 13221 catalog_manager.cc:1347] Generated new cluster ID: c102961d1bb2400c842b2a951346435e
I20260928 02:47:34.279852 13221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:34.280575 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.274689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.27.109.190:39103)
0928 02:47:34.274914 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.274923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.274957 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:47:34.275069 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.275076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.275144 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.275289 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:34.275439 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.277245 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.278378 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.278997 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.279147 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.279896 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:34.279900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:34.279900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:34.280097 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:34.280140 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:34.280154 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:34.280224 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.280300 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.280374 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260928 02:47:34.301291 13221 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:34.301993 13221 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:34.307384 13221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: Generated new TSK 0
I20260928 02:47:34.307552 13221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:34.310827 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--builtin_ntp_servers=127.27.109.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:34.389148 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:34.389369 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:34.389401 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:34.390522 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:34.390579 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:34.390619 13231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:34.391940 13231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:34.392176 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:34.392452 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:34.397408 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.397403 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.397390 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.397683 13231 server_base.cc:1034] running on GCE node
I20260928 02:47:34.397899 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:34.398144 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:34.399307 13231 hybrid_clock.cc:648] HybridClock initialized: now 1790563654399287 us; error 33 us; skew 500 ppm
I20260928 02:47:34.400614 13231 webserver.cc:460] Webserver started at http://127.27.109.129:38725/ using document root <none> and password file <none>
I20260928 02:47:34.400864 13231 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:34.400928 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:34.401060 13231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:34.402034 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "1f4f4f3ed80146afb763cbcead28fac6"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7b55f77046badf72c6917fa8f1a14fb7"
server_key_iv: "c4308a1c7cd428f87b3e40106837b57b"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.402318 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "1f4f4f3ed80146afb763cbcead28fac6"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7b55f77046badf72c6917fa8f1a14fb7"
server_key_iv: "c4308a1c7cd428f87b3e40106837b57b"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.403568 13231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:34.404234 13245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.404425 13231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:34.404511 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "1f4f4f3ed80146afb763cbcead28fac6"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "7b55f77046badf72c6917fa8f1a14fb7"
server_key_iv: "c4308a1c7cd428f87b3e40106837b57b"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.404578 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:34.425312 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:34.425561 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:34.425702 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:34.425904 13231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:34.426188 13231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:34.426221 13231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.426271 13231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:34.426308 13231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.431806 13231 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:41023
I20260928 02:47:34.431841 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:41023 every 8 connection(s)
I20260928 02:47:34.432156 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:34.435503 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13231
I20260928 02:47:34.435606 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:34.435765 28086 external_mini_cluster.cc:1428] Setting key 517fdd5a6c90f558ecbb5582db8b659d
I20260928 02:47:34.436655 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--builtin_ntp_servers=127.27.109.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:47:34.437479 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.432836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:53531 (local address 127.27.109.190:39103)
0928 02:47:34.432877 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.432879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.433148 (+ 269us) server_negotiation.cc:407] Connection header received
0928 02:47:34.433679 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.433680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.433694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.433772 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:34.434434 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.434974 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.436091 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.436291 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.436370 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.437210 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:34.437212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:34.437212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:34.437216 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:34.437231 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:34.437235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:34.437301 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.437373 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.437404 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:47:34.437677 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.432822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39103 (local address 127.27.109.129:53531)
0928 02:47:34.433015 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.433048 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.433535 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.433845 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.433849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.433864 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:34.434315 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.434319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.435069 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.435070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.436024 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.436027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.436379 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.436380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.436512 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.437048 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:34.437049 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:34.437075 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:34.437096 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:34.437303 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:34.437309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:34.437312 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.437378 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.437470 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260928 02:47:34.438215 13359 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39103
I20260928 02:47:34.438318 13359 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:34.438525 13359 heartbeater.cc:511] Master 127.27.109.190:39103 requested a full tablet report, sending...
I20260928 02:47:34.439148 13171 ts_manager.cc:194] Registered new tserver with Master: 1f4f4f3ed80146afb763cbcead28fac6 (127.27.109.129:41023)
I20260928 02:47:34.439777 13171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.129:53531
W20260928 02:47:34.516110 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:34.516301 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:34.516323 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:34.517462 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:34.517501 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:34.517526 13362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:34.518864 13362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:34.519088 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:34.519361 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:34.524333 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.524327 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.524335 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.524346 13362 server_base.cc:1034] running on GCE node
I20260928 02:47:34.524749 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:34.525013 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:34.526177 13362 hybrid_clock.cc:648] HybridClock initialized: now 1790563654526229 us; error 95 us; skew 500 ppm
I20260928 02:47:34.527457 13362 webserver.cc:460] Webserver started at http://127.27.109.130:42311/ using document root <none> and password file <none>
I20260928 02:47:34.527694 13362 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:34.527765 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:34.527899 13362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:34.528959 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "0f8dd3a92aa3a08f260656ebc1bf7f11"
server_key_iv: "b4f70e212a673dc9d2afaed76a2b3bd6"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.529310 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "0f8dd3a92aa3a08f260656ebc1bf7f11"
server_key_iv: "b4f70e212a673dc9d2afaed76a2b3bd6"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.530630 13362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:34.531373 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.531549 13362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:34.531649 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "0f8dd3a92aa3a08f260656ebc1bf7f11"
server_key_iv: "b4f70e212a673dc9d2afaed76a2b3bd6"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.531738 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:34.559011 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:34.559363 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:34.559531 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:34.559795 13362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:34.560118 13362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:34.560173 13362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.560222 13362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:34.560249 13362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.565915 13362 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:35733
I20260928 02:47:34.565958 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:35733 every 8 connection(s)
I20260928 02:47:34.566316 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:34.571187 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.567064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:43119 (local address 127.27.109.190:39103)
0928 02:47:34.567108 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.567110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.567344 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:47:34.567798 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.567799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.567811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.567863 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:34.568458 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.568941 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.569846 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.570028 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.570080 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.570940 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:34.570942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:34.570942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:34.570946 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:34.570967 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:34.570972 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:34.571032 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.571100 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.571124 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:47:34.571388 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.567049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39103 (local address 127.27.109.130:43119)
0928 02:47:34.567228 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.567261 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.567716 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.567913 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.567916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.567924 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:34.568394 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.568398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.569013 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.569014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.569771 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.569773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.570100 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.570101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.570212 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.570805 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:34.570806 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:34.570833 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:34.570858 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:34.571034 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:34.571036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:34.571038 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.571082 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.571154 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260928 02:47:34.571933 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13362
I20260928 02:47:34.572012 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:34.572059 13490 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39103
I20260928 02:47:34.572171 13490 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:34.572192 28086 external_mini_cluster.cc:1428] Setting key 25a7f98300898aa50c2c7cc1eb95553b
I20260928 02:47:34.572355 13490 heartbeater.cc:511] Master 127.27.109.190:39103 requested a full tablet report, sending...
I20260928 02:47:34.572724 13170 ts_manager.cc:194] Registered new tserver with Master: 4f4c5982acf8412aacf4a5fbf4ed5ff7 (127.27.109.130:35733)
I20260928 02:47:34.573167 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--builtin_ntp_servers=127.27.109.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:47:34.573225 13170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.130:43119
W20260928 02:47:34.654044 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:34.654268 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:34.654304 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:34.655463 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:34.655529 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:34.655570 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:34.656867 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:34.657115 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:34.657384 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:34.662086 13499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:34.662124 13498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.662279 13493 server_base.cc:1034] running on GCE node
W20260928 02:47:34.662308 13501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:34.662592 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:34.662824 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:34.663990 13493 hybrid_clock.cc:648] HybridClock initialized: now 1790563654663971 us; error 30 us; skew 500 ppm
I20260928 02:47:34.665254 13493 webserver.cc:460] Webserver started at http://127.27.109.131:41369/ using document root <none> and password file <none>
I20260928 02:47:34.665485 13493 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:34.665555 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:34.665716 13493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:34.666819 13493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "6ae3f783c84d4e2b8d4f585de6caf273"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "8fd0a9c07efbd19dcd7e5ed5a333b4cf"
server_key_iv: "81ca4023abf989f0b78ea398cec923e8"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.667160 13493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae3f783c84d4e2b8d4f585de6caf273"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "8fd0a9c07efbd19dcd7e5ed5a333b4cf"
server_key_iv: "81ca4023abf989f0b78ea398cec923e8"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.668370 13493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:34.669085 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.669265 13493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:34.669359 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "6ae3f783c84d4e2b8d4f585de6caf273"
format_stamp: "Formatted at 2026-09-28 02:47:34 on dist-test-slave-b5p5"
server_key: "8fd0a9c07efbd19dcd7e5ed5a333b4cf"
server_key_iv: "81ca4023abf989f0b78ea398cec923e8"
server_key_version: "encryptionkey@0"
I20260928 02:47:34.669440 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:34.684190 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:34.684450 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:34.684594 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:34.684804 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:34.685103 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:34.685156 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.685194 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:34.685220 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:34.690701 13493 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39799
I20260928 02:47:34.690750 13620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39799 every 8 connection(s)
I20260928 02:47:34.691042 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:34.695894 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.691679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:52855 (local address 127.27.109.190:39103)
0928 02:47:34.691747 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.691749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.691961 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:47:34.692424 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.692425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.692435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.692493 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:34.693112 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.693609 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.694603 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.694799 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.694852 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.695645 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:34.695646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:34.695647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:34.695651 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:34.695670 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:34.695675 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:34.695722 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.695812 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.695838 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:47:34.696022 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.691652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39103 (local address 127.27.109.131:52855)
0928 02:47:34.691832 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.691870 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.692340 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.692564 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.692567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.692574 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:34.693031 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.693035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.693690 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.693692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.694454 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.694456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.694872 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.694873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.694977 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.695502 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:34.695503 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:34.695529 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:34.695551 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:34.695745 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:34.695749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:34.695751 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.695805 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.695880 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260928 02:47:34.696553 13621 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39103
I20260928 02:47:34.696653 13621 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:34.696827 13621 heartbeater.cc:511] Master 127.27.109.190:39103 requested a full tablet report, sending...
I20260928 02:47:34.697181 13170 ts_manager.cc:194] Registered new tserver with Master: 6ae3f783c84d4e2b8d4f585de6caf273 (127.27.109.131:39799)
I20260928 02:47:34.697553 13170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.131:52855
I20260928 02:47:34.698154 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13493
I20260928 02:47:34.698256 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:34.698434 28086 external_mini_cluster.cc:1428] Setting key a5fa83ea54d1fbb7e75474ff89199ee5
I20260928 02:47:34.699251 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:34.703107 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.700544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.27.109.190:39103)
0928 02:47:34.700583 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.700584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.700678 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:47:34.700769 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.700769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.700778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.700814 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:34.700940 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.701370 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.702067 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.702250 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.702299 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.702938 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:34.702939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:34.702939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:34.702942 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:34.702952 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:34.702955 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:34.702995 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.703041 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.703058 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:47:34.704537 13170 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:34.706029 13170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:34.711869 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.708114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.27.109.131:39799)
0928 02:47:34.708273 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.708277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.708469 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:47:34.708502 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.708504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.708524 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.708609 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.708750 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.709504 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.710831 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.711453 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711495 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.711548 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.711756 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.711791 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260928 02:47:34.711866 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.707968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39799 (local address 127.0.0.1:45342)
0928 02:47:34.708288 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.708329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.708394 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.708612 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.708615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.708617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.708671 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.708673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.709598 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.709599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.710682 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.710691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.711578 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.711630 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.711637 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.711740 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.711789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":69,"threads_started":1}
I20260928 02:47:34.712282 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.707968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:41023 (local address 127.0.0.1:53746)
0928 02:47:34.708289 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.708330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.708396 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.708699 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.708700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.708701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.708737 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.708738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.710082 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.710083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.711053 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.711063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.712051 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.712052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.712113 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.712116 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.712166 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.712209 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":5,"thread_start_us":54,"threads_started":1}
I20260928 02:47:34.712366 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.708163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35733 (local address 127.0.0.1:42054)
0928 02:47:34.708318 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.708330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.708394 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.708737 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.708739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.708741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.708789 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.708791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.710071 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.710072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.711151 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.711153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.712081 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.712081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.712130 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.712133 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.712226 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.712256 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":32,"thread_start_us":69,"threads_started":1}
I20260928 02:47:34.712574 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.708064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.27.109.129:41023)
0928 02:47:34.708534 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.708538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.708546 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:47:34.708602 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.708604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.708618 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.708855 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.708924 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.709992 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711141 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.711861 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711935 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.712015 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.712450 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.712496 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":103,"threads_started":1}
I20260928 02:47:34.712779 13555 tablet_service.cc:1467] Processing CreateTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 (DEFAULT_TABLE table=test-table [id=3bb05206fbdb40ea881b26e969b406a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:34.713131 13555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745ba6cf53b94418a600bf0ae0ad6e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:34.713218 13293 tablet_service.cc:1467] Processing CreateTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 (DEFAULT_TABLE table=test-table [id=3bb05206fbdb40ea881b26e969b406a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:34.713513 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745ba6cf53b94418a600bf0ae0ad6e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:34.714741 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.708274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.27.109.130:35733)
0928 02:47:34.708481 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.708486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.708496 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:47:34.708540 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.708542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.708574 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.708672 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.708872 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.709981 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711247 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.711927 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.711987 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.712054 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.714573 (+ 2519us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.714645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260928 02:47:34.715248 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 (DEFAULT_TABLE table=test-table [id=3bb05206fbdb40ea881b26e969b406a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:34.715474 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745ba6cf53b94418a600bf0ae0ad6e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:34.716425 13635 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Bootstrap starting.
I20260928 02:47:34.716799 13636 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Bootstrap starting.
I20260928 02:47:34.717057 13635 tablet_bootstrap.cc:654] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:34.717381 13636 tablet_bootstrap.cc:654] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:34.717437 13635 log.cc:824] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:34.717741 13636 log.cc:824] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:34.718138 13635 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: No bootstrap required, opened a new log
I20260928 02:47:34.718227 13635 ts_tablet_manager.cc:1397] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:34.718591 13636 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: No bootstrap required, opened a new log
I20260928 02:47:34.718613 13639 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Bootstrap starting.
I20260928 02:47:34.718742 13636 ts_tablet_manager.cc:1397] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:34.719429 13639 tablet_bootstrap.cc:654] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:34.719724 13639 log.cc:824] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:34.720345 13639 tablet_bootstrap.cc:492] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: No bootstrap required, opened a new log
I20260928 02:47:34.720420 13639 ts_tablet_manager.cc:1397] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:34.720809 13635 raft_consensus.cc:348] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.720983 13635 raft_consensus.cc:374] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:34.721019 13635 raft_consensus.cc:729] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae3f783c84d4e2b8d4f585de6caf273, State: Initialized, Role: FOLLOWER
I20260928 02:47:34.721160 13635 consensus_queue.cc:260] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.721211 13636 raft_consensus.cc:348] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.721365 13636 raft_consensus.cc:374] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:34.721407 13636 raft_consensus.cc:729] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4f4f3ed80146afb763cbcead28fac6, State: Initialized, Role: FOLLOWER
I20260928 02:47:34.721427 13635 ts_tablet_manager.cc:1428] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:34.721536 13621 heartbeater.cc:503] Master 127.27.109.190:39103 was elected leader, sending a full tablet report...
I20260928 02:47:34.721516 13636 consensus_queue.cc:260] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.721787 13636 ts_tablet_manager.cc:1428] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:34.721922 13359 heartbeater.cc:503] Master 127.27.109.190:39103 was elected leader, sending a full tablet report...
I20260928 02:47:34.723539 13639 raft_consensus.cc:348] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.723703 13639 raft_consensus.cc:374] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:34.723748 13639 raft_consensus.cc:729] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4c5982acf8412aacf4a5fbf4ed5ff7, State: Initialized, Role: FOLLOWER
I20260928 02:47:34.723852 13639 consensus_queue.cc:260] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.724076 13639 ts_tablet_manager.cc:1428] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:47:34.724129 13490 heartbeater.cc:503] Master 127.27.109.190:39103 was elected leader, sending a full tablet report...
W20260928 02:47:34.816874 13491 tablet.cc:2369] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:34.839687 13641 raft_consensus.cc:482] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:34.839826 13641 raft_consensus.cc:504] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.840253 13641 leader_election.cc:288] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f4f4f3ed80146afb763cbcead28fac6 (127.27.109.129:41023), 4f4c5982acf8412aacf4a5fbf4ed5ff7 (127.27.109.130:35733)
I20260928 02:47:34.844079 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.840512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41893 (local address 127.27.109.130:35733)
0928 02:47:34.840555 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.840557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.840568 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:34.840599 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.840600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.840610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.840673 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.840853 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.841642 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843134 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.843697 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843770 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.843778 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.843965 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.844009 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:47:34.844077 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.840417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:35733 (local address 127.27.109.131:41893)
0928 02:47:34.840470 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.840483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.840529 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.840692 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.840693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.840695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.840778 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.840780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.841716 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.841717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.843016 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.843019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.843783 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.843861 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.843867 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.843925 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.843977 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":46}
I20260928 02:47:34.844095 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.840486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:38289 (local address 127.27.109.129:41023)
0928 02:47:34.840529 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.840531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.840774 (+ 243us) server_negotiation.cc:407] Connection header received
0928 02:47:34.840809 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.840827 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.840841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.840892 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:34.841022 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.841689 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843134 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.843686 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.843778 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.843965 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.844011 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:47:34.844092 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.840424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:41023 (local address 127.27.109.131:38289)
0928 02:47:34.840689 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.840698 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:34.840757 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:34.840906 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:34.840906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:34.840908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:34.840953 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.840954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.841759 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.841760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.843016 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:34.843019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.843784 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.843785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:34.843861 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.843867 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:47:34.843925 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:47:34.843977 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":70,"thread_start_us":106,"threads_started":1}
I20260928 02:47:34.844486 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745ba6cf53b94418a600bf0ae0ad6e60" candidate_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" is_pre_election: true
I20260928 02:47:34.844491 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745ba6cf53b94418a600bf0ae0ad6e60" candidate_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4f4f3ed80146afb763cbcead28fac6" is_pre_election: true
I20260928 02:47:34.844643 13444 raft_consensus.cc:2454] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae3f783c84d4e2b8d4f585de6caf273 in term 0.
I20260928 02:47:34.844643 13313 raft_consensus.cc:2454] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae3f783c84d4e2b8d4f585de6caf273 in term 0.
I20260928 02:47:34.844918 13511 leader_election.cc:302] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4f4f3ed80146afb763cbcead28fac6, 6ae3f783c84d4e2b8d4f585de6caf273; no voters:
I20260928 02:47:34.845084 13641 raft_consensus.cc:2781] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:34.845157 13641 raft_consensus.cc:482] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:34.845192 13641 raft_consensus.cc:3037] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.845880 13641 raft_consensus.cc:504] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.846024 13641 leader_election.cc:288] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [CANDIDATE]: Term 1 election: Requested vote from peers 1f4f4f3ed80146afb763cbcead28fac6 (127.27.109.129:41023), 4f4c5982acf8412aacf4a5fbf4ed5ff7 (127.27.109.130:35733)
I20260928 02:47:34.846184 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745ba6cf53b94418a600bf0ae0ad6e60" candidate_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f4f4f3ed80146afb763cbcead28fac6"
I20260928 02:47:34.846185 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745ba6cf53b94418a600bf0ae0ad6e60" candidate_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7"
I20260928 02:47:34.846274 13313 raft_consensus.cc:3037] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.846274 13444 raft_consensus.cc:3037] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:34.847245 13444 raft_consensus.cc:2454] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae3f783c84d4e2b8d4f585de6caf273 in term 1.
I20260928 02:47:34.847306 13313 raft_consensus.cc:2454] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae3f783c84d4e2b8d4f585de6caf273 in term 1.
I20260928 02:47:34.847446 13509 leader_election.cc:302] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4c5982acf8412aacf4a5fbf4ed5ff7, 6ae3f783c84d4e2b8d4f585de6caf273; no voters:
I20260928 02:47:34.847553 13641 raft_consensus.cc:2781] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:34.847699 13641 raft_consensus.cc:686] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 1 LEADER]: Becoming Leader. State: Replica: 6ae3f783c84d4e2b8d4f585de6caf273, State: Running, Role: LEADER
I20260928 02:47:34.847796 13641 consensus_queue.cc:237] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } }
I20260928 02:47:34.848608 13171 catalog_manager.cc:5495] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae3f783c84d4e2b8d4f585de6caf273 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae3f783c84d4e2b8d4f585de6caf273" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:34.854666 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.852385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.27.109.131:39799)
0928 02:47:34.852437 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:34.852439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:34.852447 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:47:34.852480 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:34.852480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:34.852502 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:34.852571 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:34.852730 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.853160 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.853878 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:34.854064 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:34.854166 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:34.854329 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:34.854420 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:34.854540 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:34.854557 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 02:47:34.854575 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:47:34.856171 13313 raft_consensus.cc:1261] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 1 FOLLOWER]: Refusing update from remote peer 6ae3f783c84d4e2b8d4f585de6caf273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:34.856171 13444 raft_consensus.cc:1261] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 1 FOLLOWER]: Refusing update from remote peer 6ae3f783c84d4e2b8d4f585de6caf273: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:34.856468 13641 consensus_queue.cc:1035] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4c5982acf8412aacf4a5fbf4ed5ff7" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:34.856688 13645 consensus_queue.cc:1035] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4f4f3ed80146afb763cbcead28fac6" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:34.858521 13648 mvcc.cc:205] Tried to move back new op lower bound from 7334148730288726016 to 7334148730257244160. Current Snapshot: MvccSnapshot[applied={T|T < 7334148730288726016}]
I20260928 02:47:34.858752 13649 mvcc.cc:205] Tried to move back new op lower bound from 7334148730288726016 to 7334148730257244160. Current Snapshot: MvccSnapshot[applied={T|T < 7334148730288726016}]
I20260928 02:47:34.860664 13170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:34.860796 13170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:34.861480 13170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a489f10e628494aad90f61a7be87af1: Sending DeleteTablet for 3 replicas of tablet 745ba6cf53b94418a600bf0ae0ad6e60
I20260928 02:47:34.861769 13555 tablet_service.cc:1514] Processing DeleteTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='slave'} at 127.0.0.1:45342
I20260928 02:47:34.861809 13293 tablet_service.cc:1514] Processing DeleteTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='slave'} at 127.0.0.1:53746
I20260928 02:47:34.861972 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 745ba6cf53b94418a600bf0ae0ad6e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:34 UTC) from {username='slave'} at 127.0.0.1:42054
I20260928 02:47:34.862015 13659 tablet_replica.cc:330] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: stopping tablet replica
I20260928 02:47:34.862105 13659 raft_consensus.cc:2229] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:34.862223 13659 raft_consensus.cc:2258] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.862411 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13231
I20260928 02:47:34.862485 13658 tablet_replica.cc:330] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: stopping tablet replica
I20260928 02:47:34.862569 13658 raft_consensus.cc:2229] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:47:34.862646 13658 raft_consensus.cc:2258] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.862668 13659 ts_tablet_manager.cc:1905] T 745ba6cf53b94418a600bf0ae0ad6e60 P 1f4f4f3ed80146afb763cbcead28fac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:34.864045 13658 ts_tablet_manager.cc:1905] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:34.865582 13658 ts_tablet_manager.cc:1918] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:34.865686 13658 log.cc:1196] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/745ba6cf53b94418a600bf0ae0ad6e60
I20260928 02:47:34.865976 13658 ts_tablet_manager.cc:1939] T 745ba6cf53b94418a600bf0ae0ad6e60 P 6ae3f783c84d4e2b8d4f585de6caf273: Deleting consensus metadata
I20260928 02:47:34.866485 13156 catalog_manager.cc:4841] TS 6ae3f783c84d4e2b8d4f585de6caf273 (127.27.109.131:39799): tablet 745ba6cf53b94418a600bf0ae0ad6e60 (table test-table [id=3bb05206fbdb40ea881b26e969b406a6]) successfully deleted
I20260928 02:47:34.866551 13660 tablet_replica.cc:330] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: stopping tablet replica
I20260928 02:47:34.866638 13660 raft_consensus.cc:2229] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:34.866739 13660 raft_consensus.cc:2258] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:34.867159 13660 ts_tablet_manager.cc:1905] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:34.868698 13660 ts_tablet_manager.cc:1918] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:34.868770 13660 log.cc:1196] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/745ba6cf53b94418a600bf0ae0ad6e60
I20260928 02:47:34.869001 13660 ts_tablet_manager.cc:1939] T 745ba6cf53b94418a600bf0ae0ad6e60 P 4f4c5982acf8412aacf4a5fbf4ed5ff7: Deleting consensus metadata
W20260928 02:47:34.869217 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:41023 (error 108)
I20260928 02:47:34.869421 13156 catalog_manager.cc:4841] TS 4f4c5982acf8412aacf4a5fbf4ed5ff7 (127.27.109.130:35733): tablet 745ba6cf53b94418a600bf0ae0ad6e60 (table test-table [id=3bb05206fbdb40ea881b26e969b406a6]) successfully deleted
I20260928 02:47:34.869613 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13362
I20260928 02:47:34.869771 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:34.869625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:41023 (local address 127.0.0.1:53758)
0928 02:47:34.869693 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:34.869736 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:47:34.870354 13158 catalog_manager.cc:4584] TS 1f4f4f3ed80146afb763cbcead28fac6 (127.27.109.129:41023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 745ba6cf53b94418a600bf0ae0ad6e60: Network error: Client connection negotiation failed: client connection to 127.27.109.129:41023: connect: Connection refused (error 111)
I20260928 02:47:34.874848 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13493
I20260928 02:47:34.880353 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13140
2026-09-28T02:47:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:34 dist-test-slave-b5p5 krb5kdc[13664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:34 dist-test-slave-b5p5 krb5kdc[13664](info): set up 2 sockets
Sep 28 02:47:34 dist-test-slave-b5p5 krb5kdc[13664](info): commencing operation
krb5kdc: starting...
W20260928 02:47:35.910233 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:35 dist-test-slave-b5p5 krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:36.019418 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39549
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:36.098083 13677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.098271 13677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.098294 13677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.099437 13677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:36.099479 13677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.099493 13677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.099506 13677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:36.099558 13677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:36.100957 13677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.101188 13677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.101423 13677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:47:36.106076 13677 server_base.cc:1034] running on GCE node
W20260928 02:47:36.106057 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.106057 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.106048 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.106561 13677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.106835 13677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.107967 13677 hybrid_clock.cc:648] HybridClock initialized: now 1790563656107957 us; error 33 us; skew 500 ppm
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:36.110381 13677 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:36.110817 13677 webserver.cc:460] Webserver started at http://127.27.109.190:39551/ using document root <none> and password file <none>
I20260928 02:47:36.111039 13677 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.111083 13677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.111220 13677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.112231 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "be99f36c1b8c419f904c2da929a27605"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "7a2d0311819a885ff0b7374150c68667"
server_key_iv: "742918a4bad147fd15fbf9e614b48140"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.112591 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "be99f36c1b8c419f904c2da929a27605"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "7a2d0311819a885ff0b7374150c68667"
server_key_iv: "742918a4bad147fd15fbf9e614b48140"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.113893 13677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:36.114640 13692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.114825 13677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:36.114897 13677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "be99f36c1b8c419f904c2da929a27605"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "7a2d0311819a885ff0b7374150c68667"
server_key_iv: "742918a4bad147fd15fbf9e614b48140"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.114977 13677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:36.126928 13677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:36.128561 13677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:36.128719 13677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:36.132524 13677 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:39549
I20260928 02:47:36.132552 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:39549 every 8 connection(s)
I20260928 02:47:36.132866 13677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:36.133363 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13677
I20260928 02:47:36.133486 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:36.133673 28086 external_mini_cluster.cc:1428] Setting key 5007293babb0a275da9d1d6b7aecac4d
I20260928 02:47:36.133868 13745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:36.136539 13745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: Bootstrap starting.
I20260928 02:47:36.137115 13745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:36.137398 13745 log.cc:824] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:36.138120 13745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: No bootstrap required, opened a new log
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:36.140355 13745 raft_consensus.cc:348] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } }
I20260928 02:47:36.140502 13745 raft_consensus.cc:374] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:36.140547 13745 raft_consensus.cc:729] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be99f36c1b8c419f904c2da929a27605, State: Initialized, Role: FOLLOWER
I20260928 02:47:36.140671 13745 consensus_queue.cc:260] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } }
I20260928 02:47:36.140785 13745 raft_consensus.cc:388] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:36.140821 13745 raft_consensus.cc:482] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:36.140872 13745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:36.141578 13745 raft_consensus.cc:504] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } }
I20260928 02:47:36.141690 13745 leader_election.cc:302] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be99f36c1b8c419f904c2da929a27605; no voters:
I20260928 02:47:36.141875 13745 leader_election.cc:288] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:36.141948 13750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:36.142112 13750 raft_consensus.cc:686] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [term 1 LEADER]: Becoming Leader. State: Replica: be99f36c1b8c419f904c2da929a27605, State: Running, Role: LEADER
I20260928 02:47:36.142143 13745 sys_catalog.cc:564] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:36.142230 13750 consensus_queue.cc:237] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } }
I20260928 02:47:36.142544 13750 sys_catalog.cc:455] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be99f36c1b8c419f904c2da929a27605. Latest consensus state: current_term: 1 leader_uuid: "be99f36c1b8c419f904c2da929a27605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } } }
I20260928 02:47:36.142621 13750 sys_catalog.cc:458] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:36.142786 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be99f36c1b8c419f904c2da929a27605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be99f36c1b8c419f904c2da929a27605" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 39549 } } }
I20260928 02:47:36.142859 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605 [sys.catalog]: This master's current role is: LEADER
W20260928 02:47:36.144023 13766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:47:36.144121 13766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:47:36.144227 13757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:36.144397 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.134143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.27.109.190:39549)
0928 02:47:36.134347 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.134354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.134377 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:47:36.134978 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.134985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.135029 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.135165 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.135804 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.136700 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.137572 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.138115 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.138246 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.140749 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.140753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.140754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:36.140765 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.142648 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.143098 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.143100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.143101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.143229 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.143472 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.143473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.143474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.143766 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.143934 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.144020 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.144154 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260928 02:47:36.144408 13757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:36.145887 13757 catalog_manager.cc:1347] Generated new cluster ID: cedd28bea28545bc8c948619871695c1
I20260928 02:47:36.145936 13757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:36.157322 13757 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:36.157909 13757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:36.160569 13757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: Generated new TSK 0
I20260928 02:47:36.160763 13757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:47:36.188472 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:36.266855 13771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.267063 13771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.267102 13771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.268208 13771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.268277 13771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.268322 13771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:36.269731 13771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.269976 13771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.270187 13771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:36.275175 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.275213 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.275188 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.275767 13771 server_base.cc:1034] running on GCE node
I20260928 02:47:36.275913 13771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.276093 13771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.277257 13771 hybrid_clock.cc:648] HybridClock initialized: now 1790563656277241 us; error 27 us; skew 500 ppm
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:36.280072 13771 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:47:36.280457 13771 webserver.cc:460] Webserver started at http://127.27.109.129:36959/ using document root <none> and password file <none>
I20260928 02:47:36.280628 13771 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.280668 13771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.280753 13771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.281685 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "25d842c8f41c434da757cd1e76fbf678"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "b89d3b49ee1967f31f226419c7e30c80"
server_key_iv: "7bc83b392a1d1d8422b2fabaf867723a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.281980 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "25d842c8f41c434da757cd1e76fbf678"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "b89d3b49ee1967f31f226419c7e30c80"
server_key_iv: "7bc83b392a1d1d8422b2fabaf867723a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.283365 13771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:36.284070 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.284253 13771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:36.284358 13771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "25d842c8f41c434da757cd1e76fbf678"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "b89d3b49ee1967f31f226419c7e30c80"
server_key_iv: "7bc83b392a1d1d8422b2fabaf867723a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.284437 13771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:36.310104 13771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:36.312124 13771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:36.312279 13771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:36.312498 13771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:36.312804 13771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:36.312853 13771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.312883 13771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:36.312912 13771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.318679 13771 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:37695
I20260928 02:47:36.318758 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:37695 every 8 connection(s)
I20260928 02:47:36.319056 13771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:36.323067 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13771
I20260928 02:47:36.323176 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:36.323380 28086 external_mini_cluster.cc:1428] Setting key 92b71163c4334dd935084e33edc926aa
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:36.328377 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.319753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:52131 (local address 127.27.109.190:39549)
0928 02:47:36.319797 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.319799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.319964 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:47:36.320404 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.320405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.320413 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.320465 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.321296 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.321782 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.322809 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.323007 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.323061 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.325422 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.325423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.325423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:36.325430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.327082 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.327449 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.327451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.327451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.327517 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.327738 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.327739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.327740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.327890 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.327968 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.328188 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.328274 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:47:36.328388 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.319684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39549 (local address 127.27.109.129:52131)
0928 02:47:36.319840 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.319878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.320326 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.320515 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.320518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.321047 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:36.321223 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.321228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.321864 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.321865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.322739 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.322742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.323091 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.323093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.323248 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.323764 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:36.323774 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:36.325308 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:36.327165 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.327170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.327172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.327377 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.327584 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.327585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.327585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.327673 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.327959 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:36.327963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:36.328063 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.328140 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.328245 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260928 02:47:36.328884 13900 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39549
I20260928 02:47:36.328990 13900 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:36.329167 13900 heartbeater.cc:511] Master 127.27.109.190:39549 requested a full tablet report, sending...
I20260928 02:47:36.329694 13709 ts_manager.cc:194] Registered new tserver with Master: 25d842c8f41c434da757cd1e76fbf678 (127.27.109.129:37695)
I20260928 02:47:36.330346 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:52131
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:47:36.349241 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:36.428331 13905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.428519 13905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.428540 13905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.429651 13905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.429688 13905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.429713 13905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:36.431046 13905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.431206 13905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.431375 13905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:36.436096 13910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.436097 13913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.436120 13911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.436358 13905 server_base.cc:1034] running on GCE node
I20260928 02:47:36.436681 13905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.436861 13905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.438004 13905 hybrid_clock.cc:648] HybridClock initialized: now 1790563656437989 us; error 27 us; skew 500 ppm
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:36.440508 13905 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:47:36.440912 13905 webserver.cc:460] Webserver started at http://127.27.109.130:33821/ using document root <none> and password file <none>
I20260928 02:47:36.441124 13905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.441188 13905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.441336 13905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.442509 13905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "5bd434c399e948a99095b2a7f242e056"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "1ce6a43e0adf11b21e040d010b1211cb"
server_key_iv: "305429fb9eba2f261453712161692538"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.442875 13905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "5bd434c399e948a99095b2a7f242e056"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "1ce6a43e0adf11b21e040d010b1211cb"
server_key_iv: "305429fb9eba2f261453712161692538"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.444276 13905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:36.444945 13920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.445118 13905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:36.445218 13905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "5bd434c399e948a99095b2a7f242e056"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "1ce6a43e0adf11b21e040d010b1211cb"
server_key_iv: "305429fb9eba2f261453712161692538"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.445307 13905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:36.468827 13905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:36.471101 13905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:36.471263 13905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:36.471530 13905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:36.471835 13905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:36.471868 13905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.471889 13905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:36.471904 13905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.477324 13905 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44181
I20260928 02:47:36.477458 14033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44181 every 8 connection(s)
I20260928 02:47:36.477659 13905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:36.483798 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 13905
I20260928 02:47:36.483886 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:36.484059 28086 external_mini_cluster.cc:1428] Setting key 36cc8e1420f53b98342e272b21383be1
I20260928 02:47:36.486944 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.478384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:55893 (local address 127.27.109.190:39549)
0928 02:47:36.478462 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.478482 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.478656 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:47:36.479102 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.479104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.479114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.479160 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.479998 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.480485 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.481449 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.481632 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.481682 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.483672 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.483674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.483674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:36.483680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.485336 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.485817 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.485818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.485818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.485874 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.486153 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.486154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.486154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.486328 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.486402 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.486733 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.486829 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:47:36.487033 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.478331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39549 (local address 127.27.109.130:55893)
0928 02:47:36.478535 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.478579 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.479026 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.479216 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.479219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.479744 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:36.479928 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.479933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.480560 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.480561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.481380 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.481382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.481742 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.481744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.481848 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.482241 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:36.482249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:36.483574 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:36.485414 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.485420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.485422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.485737 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.485943 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.485944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.485944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.486078 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.486416 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:36.486422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:36.486598 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.486688 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.486808 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260928 02:47:36.487584 14034 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39549
I20260928 02:47:36.487660 14034 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:36.487870 14034 heartbeater.cc:511] Master 127.27.109.190:39549 requested a full tablet report, sending...
I20260928 02:47:36.488227 13709 ts_manager.cc:194] Registered new tserver with Master: 5bd434c399e948a99095b2a7f242e056 (127.27.109.130:44181)
I20260928 02:47:36.488792 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:55893
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:47:36.509192 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:36.588838 14039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.589026 14039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.589049 14039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.590171 14039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.590212 14039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.590274 14039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:36.591696 14039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:39549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.591921 14039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.592146 14039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:36.596957 14044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.596941 14047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.596941 14045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.597069 14039 server_base.cc:1034] running on GCE node
I20260928 02:47:36.597245 14039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.597479 14039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.598647 14039 hybrid_clock.cc:648] HybridClock initialized: now 1790563656598638 us; error 37 us; skew 500 ppm
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:36.600908 14039 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:47:36.601305 14039 webserver.cc:460] Webserver started at http://127.27.109.131:43417/ using document root <none> and password file <none>
I20260928 02:47:36.601522 14039 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.601589 14039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.601718 14039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.602766 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "68023b4e71794687a0cd2d2a9996c345"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "d603fe3b04d0b300eeb50fa4a5da6120"
server_key_iv: "e7d2720378cd0b7d79e875884f7005f8"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.603106 14039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "68023b4e71794687a0cd2d2a9996c345"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "d603fe3b04d0b300eeb50fa4a5da6120"
server_key_iv: "e7d2720378cd0b7d79e875884f7005f8"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.604405 14039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:36.605083 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.605257 14039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:36.605350 14039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "68023b4e71794687a0cd2d2a9996c345"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "d603fe3b04d0b300eeb50fa4a5da6120"
server_key_iv: "e7d2720378cd0b7d79e875884f7005f8"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.605434 14039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:36.617230 14039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:36.619400 14039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:36.619553 14039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:36.619761 14039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:36.620047 14039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:36.620100 14039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.620138 14039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:36.620167 14039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.625784 14039 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:39389
I20260928 02:47:36.625823 14167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:39389 every 8 connection(s)
I20260928 02:47:36.626147 14039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:47:36 dist-test-slave-b5p5 krb5kdc[13664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563656, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:36.633884 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14039
I20260928 02:47:36.633996 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:36.634229 28086 external_mini_cluster.cc:1428] Setting key fc29d4112efa992ac49f258e8ff04b0a
I20260928 02:47:36.634703 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.626888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:36469 (local address 127.27.109.190:39549)
0928 02:47:36.626931 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.626933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.627110 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:47:36.627553 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.627554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.627562 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.627616 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.628439 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.628964 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.629874 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.630061 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.630122 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.632015 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.632016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.632017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:36.632024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.633269 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.633605 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.633606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.633606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.633653 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.633927 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.633928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.633928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.634087 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.634149 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.634509 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.634605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:47:36.634898 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.626846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:39549 (local address 127.27.109.131:36469)
0928 02:47:36.626993 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.627032 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.627477 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.627665 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.627668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.628184 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:36.628365 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.628369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.629038 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.629038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.629786 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.629788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.630136 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.630137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.630227 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.630639 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:36.630647 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:36.631923 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:36.633338 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.633343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.633345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.633543 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.633722 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:36.633723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:36.633724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:36.633859 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:36.634169 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:36.634176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:36.634341 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.634447 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.634663 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 02:47:36.635412 14168 heartbeater.cc:347] Connected to a master server at 127.27.109.190:39549
I20260928 02:47:36.635484 14168 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:36.635602 14168 heartbeater.cc:511] Master 127.27.109.190:39549 requested a full tablet report, sending...
I20260928 02:47:36.635895 13709 ts_manager.cc:194] Registered new tserver with Master: 68023b4e71794687a0cd2d2a9996c345 (127.27.109.131:39389)
I20260928 02:47:36.636269 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:36469
I20260928 02:47:36.645907 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:36.653151 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.647244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.27.109.190:39549)
0928 02:47:36.647314 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.647316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.647404 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:47:36.647490 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.647491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.647501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.647562 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.648135 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.648582 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.649408 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.649580 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.649630 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.650831 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.650832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.650832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:36.650838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.652168 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.652495 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.652496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.652496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.652543 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:36.652697 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:36.652697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:36.652698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:36.652811 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.652886 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.652982 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.653026 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:47:36.654650 13709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:36.656136 13709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:36.662206 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39389 (local address 127.0.0.1:33062)
0928 02:47:36.658825 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.658834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.658876 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.659120 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.659120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.659122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:36.659166 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.659167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.660356 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.660357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.661233 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.661235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.661892 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.661893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.661980 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.661993 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.662080 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.662134 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260928 02:47:36.662216 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.27.109.131:39389)
0928 02:47:36.658557 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.658562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.658933 (+ 371us) server_negotiation.cc:407] Connection header received
0928 02:47:36.658971 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.658973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.658987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.659053 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:36.659247 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.660255 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.661326 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.661804 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.661894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.661951 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.662107 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.662143 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260928 02:47:36.662519 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:44181 (local address 127.0.0.1:48728)
0928 02:47:36.658441 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.658494 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.658560 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.658777 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.658780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.658783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:36.658866 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.658868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.660171 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.660172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.661230 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.661233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.662202 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.662203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.662290 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.662294 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.662350 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.662397 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":48,"threads_started":1}
I20260928 02:47:36.662645 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.27.109.129:37695)
0928 02:47:36.658610 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.658614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.658732 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:47:36.658817 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.658820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.658838 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.658913 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:36.659122 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.659842 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.661358 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.662045 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.662117 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.662165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.662513 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.662563 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260928 02:47:36.662845 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.27.109.130:44181)
0928 02:47:36.658591 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.658596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.658607 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:36.658644 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.658646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.658659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.658771 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:36.658952 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.660067 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.661339 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.662046 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.662117 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.662165 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.662711 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.662753 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":77,"threads_started":1}
I20260928 02:47:36.663043 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.658195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37695 (local address 127.0.0.1:37424)
0928 02:47:36.658660 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.658668 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.658740 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.658994 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.658995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.658997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:36.659045 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.659046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.659942 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.659943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.661274 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.661276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.662202 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.662203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.662406 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.662409 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.662487 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.662519 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":142,"thread_start_us":44,"threads_started":1}
I20260928 02:47:36.663074 14102 tablet_service.cc:1467] Processing CreateTablet for tablet 47d9d293084b483ea64691845779f277 (DEFAULT_TABLE table=test-table [id=458e058b97b04721be40a826364c4bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:36.663426 14102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d9d293084b483ea64691845779f277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:36.663482 13968 tablet_service.cc:1467] Processing CreateTablet for tablet 47d9d293084b483ea64691845779f277 (DEFAULT_TABLE table=test-table [id=458e058b97b04721be40a826364c4bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:36.663681 13834 tablet_service.cc:1467] Processing CreateTablet for tablet 47d9d293084b483ea64691845779f277 (DEFAULT_TABLE table=test-table [id=458e058b97b04721be40a826364c4bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:36.663767 13968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d9d293084b483ea64691845779f277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:36.663918 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d9d293084b483ea64691845779f277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:36.666443 14182 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: Bootstrap starting.
I20260928 02:47:36.666612 14183 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Bootstrap starting.
I20260928 02:47:36.666850 14184 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Bootstrap starting.
I20260928 02:47:36.667266 14183 tablet_bootstrap.cc:654] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:36.667274 14182 tablet_bootstrap.cc:654] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:36.667443 14184 tablet_bootstrap.cc:654] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:36.667892 14182 log.cc:824] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:36.667892 14183 log.cc:824] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:36.667925 14184 log.cc:824] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:36.668588 14183 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: No bootstrap required, opened a new log
I20260928 02:47:36.668622 14182 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: No bootstrap required, opened a new log
I20260928 02:47:36.668676 14183 ts_tablet_manager.cc:1397] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:36.668705 14182 ts_tablet_manager.cc:1397] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:47:36.670213 14183 raft_consensus.cc:348] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.670347 14183 raft_consensus.cc:374] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:36.670370 14183 raft_consensus.cc:729] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68023b4e71794687a0cd2d2a9996c345, State: Initialized, Role: FOLLOWER
I20260928 02:47:36.670244 14182 raft_consensus.cc:348] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.670401 14182 raft_consensus.cc:374] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:36.670449 14182 raft_consensus.cc:729] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d842c8f41c434da757cd1e76fbf678, State: Initialized, Role: FOLLOWER
I20260928 02:47:36.670518 14183 consensus_queue.cc:260] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.670600 14182 consensus_queue.cc:260] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.670794 14183 ts_tablet_manager.cc:1428] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:36.670855 14184 tablet_bootstrap.cc:492] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: No bootstrap required, opened a new log
I20260928 02:47:36.670871 14182 ts_tablet_manager.cc:1428] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:36.670920 14184 ts_tablet_manager.cc:1397] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:47:36.670938 13900 heartbeater.cc:503] Master 127.27.109.190:39549 was elected leader, sending a full tablet report...
I20260928 02:47:36.671178 14168 heartbeater.cc:503] Master 127.27.109.190:39549 was elected leader, sending a full tablet report...
I20260928 02:47:36.672691 14184 raft_consensus.cc:348] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.672816 14184 raft_consensus.cc:374] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:36.672837 14184 raft_consensus.cc:729] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd434c399e948a99095b2a7f242e056, State: Initialized, Role: FOLLOWER
I20260928 02:47:36.672958 14184 consensus_queue.cc:260] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.673199 14184 ts_tablet_manager.cc:1428] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:47:36.673255 14034 heartbeater.cc:503] Master 127.27.109.190:39549 was elected leader, sending a full tablet report...
I20260928 02:47:36.674840 14189 raft_consensus.cc:482] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:36.674903 14189 raft_consensus.cc:504] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.675117 14189 leader_election.cc:288] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bd434c399e948a99095b2a7f242e056 (127.27.109.130:44181), 68023b4e71794687a0cd2d2a9996c345 (127.27.109.131:39389)
I20260928 02:47:36.677798 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.675274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:39389 (local address 127.27.109.129:41719)
0928 02:47:36.675356 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.675368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.675422 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.675560 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.675561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.675562 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:36.675622 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.675623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.676249 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.676250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.676973 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.676975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.677543 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.677625 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.677630 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.677681 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.677720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:47:36.677803 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.675310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:41719 (local address 127.27.109.131:39389)
0928 02:47:36.675379 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.675381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.675409 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:47:36.675489 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.675490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.675499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.675547 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:36.675690 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.676177 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677055 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.677469 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677522 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.677527 (+ 5us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.677696 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.677732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:47:36.677894 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.675274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:44181 (local address 127.27.109.129:60673)
0928 02:47:36.675453 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.675464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:36.675524 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:36.675634 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:36.675635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:36.675636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:36.675713 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.675715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.676323 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.676323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.677088 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:36.677090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.677627 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:36.677711 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.677715 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:36.677779 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:47:36.677812 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":24,"thread_start_us":86,"threads_started":1}
I20260928 02:47:36.677896 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.675307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:60673 (local address 127.27.109.130:44181)
0928 02:47:36.675347 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.675348 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.675529 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:47:36.675570 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.675570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.675578 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.675627 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:36.675810 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.676234 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677167 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.677545 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.677614 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.677620 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.677795 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.677833 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:47:36.678102 14122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d9d293084b483ea64691845779f277" candidate_uuid: "25d842c8f41c434da757cd1e76fbf678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68023b4e71794687a0cd2d2a9996c345" is_pre_election: true
I20260928 02:47:36.678215 14122 raft_consensus.cc:2454] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d842c8f41c434da757cd1e76fbf678 in term 0.
I20260928 02:47:36.678323 13988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d9d293084b483ea64691845779f277" candidate_uuid: "25d842c8f41c434da757cd1e76fbf678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd434c399e948a99095b2a7f242e056" is_pre_election: true
I20260928 02:47:36.678412 13988 raft_consensus.cc:2454] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d842c8f41c434da757cd1e76fbf678 in term 0.
I20260928 02:47:36.678402 13788 leader_election.cc:302] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d842c8f41c434da757cd1e76fbf678, 68023b4e71794687a0cd2d2a9996c345; no voters:
I20260928 02:47:36.678532 14189 raft_consensus.cc:2781] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:36.678628 14189 raft_consensus.cc:482] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:36.678663 14189 raft_consensus.cc:3037] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:36.679404 14189 raft_consensus.cc:504] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.679536 14189 leader_election.cc:288] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [CANDIDATE]: Term 1 election: Requested vote from peers 5bd434c399e948a99095b2a7f242e056 (127.27.109.130:44181), 68023b4e71794687a0cd2d2a9996c345 (127.27.109.131:39389)
I20260928 02:47:36.679677 13988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d9d293084b483ea64691845779f277" candidate_uuid: "25d842c8f41c434da757cd1e76fbf678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd434c399e948a99095b2a7f242e056"
I20260928 02:47:36.679725 14122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d9d293084b483ea64691845779f277" candidate_uuid: "25d842c8f41c434da757cd1e76fbf678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68023b4e71794687a0cd2d2a9996c345"
I20260928 02:47:36.679759 13988 raft_consensus.cc:3037] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:36.679821 14122 raft_consensus.cc:3037] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:36.680390 13988 raft_consensus.cc:2454] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d842c8f41c434da757cd1e76fbf678 in term 1.
I20260928 02:47:36.680397 14122 raft_consensus.cc:2454] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d842c8f41c434da757cd1e76fbf678 in term 1.
I20260928 02:47:36.680554 13790 leader_election.cc:302] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d842c8f41c434da757cd1e76fbf678, 5bd434c399e948a99095b2a7f242e056; no voters:
I20260928 02:47:36.680665 14189 raft_consensus.cc:2781] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:36.680799 14189 raft_consensus.cc:686] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [term 1 LEADER]: Becoming Leader. State: Replica: 25d842c8f41c434da757cd1e76fbf678, State: Running, Role: LEADER
I20260928 02:47:36.680895 14189 consensus_queue.cc:237] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } }
I20260928 02:47:36.681555 13708 catalog_manager.cc:5495] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d842c8f41c434da757cd1e76fbf678 (127.27.109.129). New cstate: current_term: 1 leader_uuid: "25d842c8f41c434da757cd1e76fbf678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d842c8f41c434da757cd1e76fbf678" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:36.689064 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.686973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.27.109.129:37695)
0928 02:47:36.687009 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.687010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.687043 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:47:36.687097 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.687097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.687110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.687160 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:36.687257 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.687716 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.688472 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.688652 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.688701 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.688822 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:36.688893 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:36.688956 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.688999 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.689006 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:47:36.690495 14122 raft_consensus.cc:1261] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 1 FOLLOWER]: Refusing update from remote peer 25d842c8f41c434da757cd1e76fbf678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:36.690497 13988 raft_consensus.cc:1261] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 1 FOLLOWER]: Refusing update from remote peer 25d842c8f41c434da757cd1e76fbf678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:36.690809 14189 consensus_queue.cc:1035] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68023b4e71794687a0cd2d2a9996c345" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:36.690954 14192 consensus_queue.cc:1035] T 47d9d293084b483ea64691845779f277 P 25d842c8f41c434da757cd1e76fbf678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd434c399e948a99095b2a7f242e056" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:36.693249 14195 mvcc.cc:205] Tried to move back new op lower bound from 7334148737801863168 to 7334148737765441536. Current Snapshot: MvccSnapshot[applied={T|T < 7334148737801863168}]
I20260928 02:47:36.694237 14196 mvcc.cc:205] Tried to move back new op lower bound from 7334148737801863168 to 7334148737765441536. Current Snapshot: MvccSnapshot[applied={T|T < 7334148737801863168}]
I20260928 02:47:36.695415 13709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:36.695483 13709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:36.696100 13709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be99f36c1b8c419f904c2da929a27605: Sending DeleteTablet for 3 replicas of tablet 47d9d293084b483ea64691845779f277
I20260928 02:47:36.696398 13968 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d9d293084b483ea64691845779f277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:36 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:48728
I20260928 02:47:36.696434 13834 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d9d293084b483ea64691845779f277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:36 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:37424
I20260928 02:47:36.696599 14205 tablet_replica.cc:330] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: stopping tablet replica
I20260928 02:47:36.696599 14102 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d9d293084b483ea64691845779f277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:36 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:33062
I20260928 02:47:36.696696 14205 raft_consensus.cc:2229] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:36.696794 14207 tablet_replica.cc:330] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: stopping tablet replica
I20260928 02:47:36.696797 14205 raft_consensus.cc:2258] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:36.696872 14207 raft_consensus.cc:2229] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:36.696952 14207 raft_consensus.cc:2258] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:36.696995 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13771
I20260928 02:47:36.697237 14205 ts_tablet_manager.cc:1905] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:36.697312 14207 ts_tablet_manager.cc:1905] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:36.698902 14205 ts_tablet_manager.cc:1918] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:36.698902 14207 ts_tablet_manager.cc:1918] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:36.698964 14205 log.cc:1196] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/47d9d293084b483ea64691845779f277
I20260928 02:47:36.699009 14207 log.cc:1196] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/47d9d293084b483ea64691845779f277
I20260928 02:47:36.699256 14205 ts_tablet_manager.cc:1939] T 47d9d293084b483ea64691845779f277 P 5bd434c399e948a99095b2a7f242e056: Deleting consensus metadata
I20260928 02:47:36.699271 14207 ts_tablet_manager.cc:1939] T 47d9d293084b483ea64691845779f277 P 68023b4e71794687a0cd2d2a9996c345: Deleting consensus metadata
I20260928 02:47:36.699609 13694 catalog_manager.cc:4841] TS 68023b4e71794687a0cd2d2a9996c345 (127.27.109.131:39389): tablet 47d9d293084b483ea64691845779f277 (table test-table [id=458e058b97b04721be40a826364c4bdc]) successfully deleted
I20260928 02:47:36.699609 13696 catalog_manager.cc:4841] TS 5bd434c399e948a99095b2a7f242e056 (127.27.109.130:44181): tablet 47d9d293084b483ea64691845779f277 (table test-table [id=458e058b97b04721be40a826364c4bdc]) successfully deleted
W20260928 02:47:36.701848 13693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:37695 (error 108)
I20260928 02:47:36.702252 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.702142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:37695 (local address 127.0.0.1:37448)
0928 02:47:36.702191 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:36.702230 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:47:36.702368 13693 catalog_manager.cc:4584] TS 25d842c8f41c434da757cd1e76fbf678 (127.27.109.129:37695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47d9d293084b483ea64691845779f277: Network error: Client connection negotiation failed: client connection to 127.27.109.129:37695: connect: Connection refused (error 111)
I20260928 02:47:36.702764 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13905
I20260928 02:47:36.708178 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14039
I20260928 02:47:36.712443 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 13677
2026-09-28T02:47:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1833 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:36Z Disabled control of system clock
I20260928 02:47:36.739207 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35365
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:36.816650 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.816862 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.816882 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.817936 14212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:36.817970 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.817981 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.817994 14212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:36.818006 14212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:36.819361 14212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:35365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.819541 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.819767 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:36.824209 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.824373 14212 server_base.cc:1034] running on GCE node
W20260928 02:47:36.824234 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.824234 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.824684 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.824973 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.826134 14212 hybrid_clock.cc:648] HybridClock initialized: now 1790563656826117 us; error 47 us; skew 500 ppm
I20260928 02:47:36.827284 14212 webserver.cc:460] Webserver started at http://127.27.109.190:33203/ using document root <none> and password file <none>
I20260928 02:47:36.827514 14212 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.827582 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.827728 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.828733 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "0e1e113eeacd43c4993d13152cdae11d"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "39002fe3e1f87ab00e15fbeb9bf6cbb9"
server_key_iv: "06188ab2e19732f0a42630b9358e931a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.829082 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "0e1e113eeacd43c4993d13152cdae11d"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "39002fe3e1f87ab00e15fbeb9bf6cbb9"
server_key_iv: "06188ab2e19732f0a42630b9358e931a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.830276 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:36.831060 14226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.831230 14212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.831327 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "0e1e113eeacd43c4993d13152cdae11d"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "39002fe3e1f87ab00e15fbeb9bf6cbb9"
server_key_iv: "06188ab2e19732f0a42630b9358e931a"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.831410 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:36.853618 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:36.853895 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:36.854038 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:36.857681 14212 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:35365
I20260928 02:47:36.857717 14278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:35365 every 8 connection(s)
I20260928 02:47:36.858058 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:36.858973 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:36.861155 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: Bootstrap starting.
I20260928 02:47:36.861714 14279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:36.862008 14279 log.cc:824] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:36.862728 14279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: No bootstrap required, opened a new log
I20260928 02:47:36.862880 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14212
I20260928 02:47:36.863000 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:36.863166 28086 external_mini_cluster.cc:1428] Setting key 132a05c9cbd2509a243fd1c1b1dce193
I20260928 02:47:36.864961 14279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } }
I20260928 02:47:36.865131 14279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:36.865167 14279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1e113eeacd43c4993d13152cdae11d, State: Initialized, Role: FOLLOWER
I20260928 02:47:36.865289 14279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } }
I20260928 02:47:36.865393 14279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:36.865425 14279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:36.865463 14279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:36.866153 14279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } }
I20260928 02:47:36.866261 14279 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e1e113eeacd43c4993d13152cdae11d; no voters:
I20260928 02:47:36.866415 14279 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:36.866500 14284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:36.866636 14284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [term 1 LEADER]: Becoming Leader. State: Replica: 0e1e113eeacd43c4993d13152cdae11d, State: Running, Role: LEADER
I20260928 02:47:36.866669 14279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:36.866716 14284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } }
I20260928 02:47:36.867023 14285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e1e113eeacd43c4993d13152cdae11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } } }
I20260928 02:47:36.867013 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1e113eeacd43c4993d13152cdae11d. Latest consensus state: current_term: 1 leader_uuid: "0e1e113eeacd43c4993d13152cdae11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1e113eeacd43c4993d13152cdae11d" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35365 } } }
I20260928 02:47:36.867180 14285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:36.867190 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:36.867951 14293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:36.868490 14293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:36.869997 14293 catalog_manager.cc:1347] Generated new cluster ID: 9b8621608e9c4a5fb11e49d9dd55cc66
I20260928 02:47:36.870061 14293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:36.870723 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:36.863625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.27.109.190:35365)
0928 02:47:36.863846 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:36.863854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:36.863885 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:47:36.864927 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:36.864935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:36.865013 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:36.865155 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:36.865394 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.867173 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.868410 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:36.869016 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:36.869125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:36.869958 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:36.869962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:36.869963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:36.870157 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:36.870199 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:36.870213 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:36.870271 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:36.870437 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:47:36.870529 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260928 02:47:36.881255 14293 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:36.881708 14293 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:36.892477 14293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: Generated new TSK 0
I20260928 02:47:36.892645 14293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:36.897464 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--builtin_ntp_servers=127.27.109.148:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:36.975526 14303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:36.975709 14303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:36.975730 14303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:36.976833 14303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:36.976871 14303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:36.976897 14303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:36.978214 14303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:36.978441 14303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:36.978742 14303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:36.983948 14308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.983948 14309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:36.984341 14311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:36.985313 14303 server_base.cc:1034] running on GCE node
I20260928 02:47:36.985526 14303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:36.985749 14303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:36.986920 14303 hybrid_clock.cc:648] HybridClock initialized: now 1790563656986896 us; error 33 us; skew 500 ppm
I20260928 02:47:36.988188 14303 webserver.cc:460] Webserver started at http://127.27.109.129:41305/ using document root <none> and password file <none>
I20260928 02:47:36.988430 14303 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:36.988495 14303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:36.988636 14303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:36.989639 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "8a381ec4fad64bb5a649c02a4b2992b3"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "2744837cdbf0306ffba9aa0d4c04c0d1"
server_key_iv: "c1232a8a7c9e4c88a2dd0a265026bcf1"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.989985 14303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "8a381ec4fad64bb5a649c02a4b2992b3"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "2744837cdbf0306ffba9aa0d4c04c0d1"
server_key_iv: "c1232a8a7c9e4c88a2dd0a265026bcf1"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.991334 14303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:36.991988 14317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:36.992137 14303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:36.992231 14303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "8a381ec4fad64bb5a649c02a4b2992b3"
format_stamp: "Formatted at 2026-09-28 02:47:36 on dist-test-slave-b5p5"
server_key: "2744837cdbf0306ffba9aa0d4c04c0d1"
server_key_iv: "c1232a8a7c9e4c88a2dd0a265026bcf1"
server_key_version: "encryptionkey@0"
I20260928 02:47:36.992322 14303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:37.016821 14303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:37.017117 14303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:37.017268 14303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:37.017479 14303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:37.017789 14303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:37.017843 14303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.017884 14303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:37.017911 14303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.023787 14303 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:45815
I20260928 02:47:37.023833 14430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:45815 every 8 connection(s)
I20260928 02:47:37.024128 14303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:37.028784 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.024758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:52895 (local address 127.27.109.190:35365)
0928 02:47:37.024817 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.024819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.025040 (+ 221us) server_negotiation.cc:407] Connection header received
0928 02:47:37.025476 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.025477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.025488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.025537 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:37.026128 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.026662 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.027581 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.027774 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.027825 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.028544 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:37.028545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:37.028545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:37.028550 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:37.028561 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:37.028564 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:37.028620 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.028696 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.028717 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:47:37.028972 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.024754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35365 (local address 127.27.109.129:52895)
0928 02:47:37.024923 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.024955 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.025395 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.025585 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.025588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.025595 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:37.026058 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.026062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.026734 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.026735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.027492 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.027494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.027843 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.027844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.027945 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.028422 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:37.028422 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:37.028447 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:37.028468 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:37.028625 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:37.028629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:37.028631 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.028681 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.028753 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260928 02:47:37.029518 14431 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35365
I20260928 02:47:37.029606 14431 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:37.029739 14431 heartbeater.cc:511] Master 127.27.109.190:35365 requested a full tablet report, sending...
I20260928 02:47:37.030201 14243 ts_manager.cc:194] Registered new tserver with Master: 8a381ec4fad64bb5a649c02a4b2992b3 (127.27.109.129:45815)
I20260928 02:47:37.030877 14243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.129:52895
I20260928 02:47:37.032262 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14303
I20260928 02:47:37.032369 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:37.032549 28086 external_mini_cluster.cc:1428] Setting key 0d6ea956f1da1a45d1838027662eeafb
I20260928 02:47:37.033306 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--builtin_ntp_servers=127.27.109.148:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:37.112111 14434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:37.112327 14434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:37.112349 14434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:37.113421 14434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:37.113456 14434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:37.113479 14434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:37.114784 14434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:37.114954 14434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:37.115170 14434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:37.119894 14440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:37.119868 14439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:37.119879 14442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:37.119999 14434 server_base.cc:1034] running on GCE node
I20260928 02:47:37.120203 14434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:37.120440 14434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:37.121574 14434 hybrid_clock.cc:648] HybridClock initialized: now 1790563657121565 us; error 25 us; skew 500 ppm
I20260928 02:47:37.122881 14434 webserver.cc:460] Webserver started at http://127.27.109.130:39659/ using document root <none> and password file <none>
I20260928 02:47:37.123127 14434 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:37.123174 14434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:37.123342 14434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:37.124331 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "65ee947139914386b285f0f3096ca934"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "2ab7c56f686bf133f65ebb4ec646bf27"
server_key_iv: "7406ef55b14ded9b20beaef760fd6a11"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.124668 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "65ee947139914386b285f0f3096ca934"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "2ab7c56f686bf133f65ebb4ec646bf27"
server_key_iv: "7406ef55b14ded9b20beaef760fd6a11"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.125880 14434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:47:37.126603 14448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.126749 14434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.126837 14434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "65ee947139914386b285f0f3096ca934"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "2ab7c56f686bf133f65ebb4ec646bf27"
server_key_iv: "7406ef55b14ded9b20beaef760fd6a11"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.126889 14434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:37.137768 14434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:37.137969 14434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:37.138057 14434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:37.138200 14434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:37.138525 14434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:37.138569 14434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.138615 14434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:37.138643 14434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.143981 14434 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:44963
I20260928 02:47:37.144035 14561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:44963 every 8 connection(s)
I20260928 02:47:37.144299 14434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:37.147655 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14434
I20260928 02:47:37.147756 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:37.147933 28086 external_mini_cluster.cc:1428] Setting key 009def454241db19dc749164ec6c950d
I20260928 02:47:37.148903 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--builtin_ntp_servers=127.27.109.148:43243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:47:37.152784 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.144999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:42203 (local address 127.27.109.190:35365)
0928 02:47:37.145060 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.145062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.145242 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:47:37.145653 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.145653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.145662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.145719 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:37.146288 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.146847 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.147930 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.148239 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.148301 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.152455 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:37.152457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:37.152458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:37.152463 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:37.152480 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:37.152485 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:37.152549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.152632 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.152666 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:47:37.152829 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.144969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35365 (local address 127.27.109.130:42203)
0928 02:47:37.145126 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.145162 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.145576 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.145766 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.145769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.145776 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:37.146220 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.146223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.146920 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.146921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.147840 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.147843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.148498 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.148500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.148681 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.152319 (+ 3638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:37.152320 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:37.152345 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:37.152365 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:37.152554 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:37.152558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:37.152559 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.152610 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.152682 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260928 02:47:37.153409 14562 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35365
I20260928 02:47:37.153508 14562 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:37.153687 14562 heartbeater.cc:511] Master 127.27.109.190:35365 requested a full tablet report, sending...
I20260928 02:47:37.154071 14243 ts_manager.cc:194] Registered new tserver with Master: 65ee947139914386b285f0f3096ca934 (127.27.109.130:44963)
I20260928 02:47:37.154438 14243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.130:42203
W20260928 02:47:37.227185 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:37.227393 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:37.227428 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:37.228613 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:37.228677 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:37.228724 14565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:37.230177 14565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:35365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:37.230406 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:37.230710 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:37.235497 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:37.235507 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:37.235510 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:37.235708 14565 server_base.cc:1034] running on GCE node
I20260928 02:47:37.235934 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:37.236184 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:37.237345 14565 hybrid_clock.cc:648] HybridClock initialized: now 1790563657237334 us; error 31 us; skew 500 ppm
I20260928 02:47:37.238648 14565 webserver.cc:460] Webserver started at http://127.27.109.131:41959/ using document root <none> and password file <none>
I20260928 02:47:37.238912 14565 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:37.238957 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:37.239099 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:37.240069 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "c33d4bc069c2453298326a9fb76d8542"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "a84233c164b6bfcb9305758511ed0a07"
server_key_iv: "5093e48d8d321501dffba945c7000832"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.240402 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "c33d4bc069c2453298326a9fb76d8542"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "a84233c164b6bfcb9305758511ed0a07"
server_key_iv: "5093e48d8d321501dffba945c7000832"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.241616 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:37.242282 14579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.242443 14565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:37.242548 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "c33d4bc069c2453298326a9fb76d8542"
format_stamp: "Formatted at 2026-09-28 02:47:37 on dist-test-slave-b5p5"
server_key: "a84233c164b6bfcb9305758511ed0a07"
server_key_iv: "5093e48d8d321501dffba945c7000832"
server_key_version: "encryptionkey@0"
I20260928 02:47:37.242602 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:37.256107 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:37.256313 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:37.256404 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:37.256561 14565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:37.256805 14565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:37.256840 14565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.256860 14565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:37.256873 14565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:37.262089 14565 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:41649
I20260928 02:47:37.262143 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:41649 every 8 connection(s)
I20260928 02:47:37.262457 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:37.263128 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14565
I20260928 02:47:37.263253 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:37.263440 28086 external_mini_cluster.cc:1428] Setting key 826819eb4e9c95e1b92f5faf3bc7202d
I20260928 02:47:37.267797 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.263212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:53947 (local address 127.27.109.190:35365)
0928 02:47:37.263267 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.263269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.263515 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:47:37.264141 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.264143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.264156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.264202 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:37.264846 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.265351 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.266310 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.266606 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.266707 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.267545 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:37.267546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:37.267547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:37.267552 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:37.267567 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:37.267572 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:37.267622 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.267691 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.267720 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:47:37.267998 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.263188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:35365 (local address 127.27.109.131:53947)
0928 02:47:37.263381 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.263418 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.264035 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.264277 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.264280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.264289 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:37.264748 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.264752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.265441 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.265442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.266212 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.266214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.266699 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.266700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.266866 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.267396 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:37.267397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:37.267424 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:37.267445 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:37.267621 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:37.267628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:37.267630 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.267694 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.267796 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260928 02:47:37.268405 14693 heartbeater.cc:347] Connected to a master server at 127.27.109.190:35365
I20260928 02:47:37.268496 14693 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:37.268671 14693 heartbeater.cc:511] Master 127.27.109.190:35365 requested a full tablet report, sending...
I20260928 02:47:37.269029 14243 ts_manager.cc:194] Registered new tserver with Master: c33d4bc069c2453298326a9fb76d8542 (127.27.109.131:41649)
I20260928 02:47:37.269408 14243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.131:53947
I20260928 02:47:37.275058 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:37.278944 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.276386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38990 (local address 127.27.109.190:35365)
0928 02:47:37.276455 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.276457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.276521 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:47:37.276561 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.276562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.276571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.276608 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:37.276759 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.277187 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.277864 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.278027 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.278072 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.278677 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:37.278678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:37.278679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:37.278682 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:37.278693 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:37.278696 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:37.278765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.278828 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.278836 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:47:37.280387 14243 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:37.284484 14243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:37.291468 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.287063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.27.109.131:41649)
0928 02:47:37.287260 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.287264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.287615 (+ 351us) server_negotiation.cc:407] Connection header received
0928 02:47:37.287656 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.287659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.287673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.287738 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:37.287951 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.288953 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.290357 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.291020 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.291087 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.291155 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.291355 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.291391 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260928 02:47:37.291461 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.286930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:41649 (local address 127.0.0.1:34030)
0928 02:47:37.287512 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.287520 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.287561 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.287814 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.287816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.287818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:37.287879 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.287880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.289046 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.289047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.290263 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.290266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.291183 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.291183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.291238 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.291246 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.291340 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.291393 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260928 02:47:37.292152 14627 tablet_service.cc:1467] Processing CreateTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 (DEFAULT_TABLE table=test-table [id=ace1dde230194566b94a3a5379b0fbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:37.292477 14627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bdc63bbf1547408ee2dca337c36a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:37.292693 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.286930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:45815 (local address 127.0.0.1:50096)
0928 02:47:37.287368 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.287378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.287417 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.287833 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.287834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.287838 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:37.287881 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.287883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.289572 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.289573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.290660 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.290663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.292404 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.292405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.292487 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.292491 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.292552 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.292599 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":89,"threads_started":1}
I20260928 02:47:37.293037 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.287222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.27.109.129:45815)
0928 02:47:37.287391 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.287395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.287410 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:47:37.287525 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.287526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.287669 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.287762 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:37.288187 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.289264 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.291012 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.291988 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.292066 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.292362 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.292883 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.292934 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260928 02:47:37.294049 14365 tablet_service.cc:1467] Processing CreateTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 (DEFAULT_TABLE table=test-table [id=ace1dde230194566b94a3a5379b0fbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:37.294412 14365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bdc63bbf1547408ee2dca337c36a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:37.295357 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.286926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:44963 (local address 127.0.0.1:44286)
0928 02:47:37.287160 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.287197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.287262 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.291820 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.291821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.291823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:37.291859 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.291861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.293092 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.293093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.294135 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.294138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.295152 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.295153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.295217 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.295221 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.295252 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.295292 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":53,"threads_started":1}
I20260928 02:47:37.295747 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.287163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.27.109.130:44963)
0928 02:47:37.291636 (+ 4473us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.291641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.291651 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:47:37.291689 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.291691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.291707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.291927 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:37.291990 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.292992 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.294237 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.295054 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.295512 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.295597 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.295620 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.295665 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4409,"thread_start_us":82,"threads_started":1}
I20260928 02:47:37.296099 14707 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Bootstrap starting.
I20260928 02:47:37.296567 14707 tablet_bootstrap.cc:654] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:37.296890 14707 log.cc:824] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:37.297559 14707 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: No bootstrap required, opened a new log
I20260928 02:47:37.297600 14707 ts_tablet_manager.cc:1397] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:37.297878 14709 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: Bootstrap starting.
I20260928 02:47:37.298179 14496 tablet_service.cc:1467] Processing CreateTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 (DEFAULT_TABLE table=test-table [id=ace1dde230194566b94a3a5379b0fbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:37.298410 14496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56bdc63bbf1547408ee2dca337c36a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:37.298560 14709 tablet_bootstrap.cc:654] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:37.298835 14709 log.cc:824] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:37.299564 14709 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: No bootstrap required, opened a new log
I20260928 02:47:37.299616 14709 ts_tablet_manager.cc:1397] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:47:37.300105 14707 raft_consensus.cc:348] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.300282 14707 raft_consensus.cc:374] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:37.300319 14707 raft_consensus.cc:729] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33d4bc069c2453298326a9fb76d8542, State: Initialized, Role: FOLLOWER
I20260928 02:47:37.300446 14707 consensus_queue.cc:260] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.300681 14707 ts_tablet_manager.cc:1428] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:37.300746 14693 heartbeater.cc:503] Master 127.27.109.190:35365 was elected leader, sending a full tablet report...
I20260928 02:47:37.300870 14712 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Bootstrap starting.
I20260928 02:47:37.301568 14712 tablet_bootstrap.cc:654] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:37.302001 14712 log.cc:824] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:37.301998 14709 raft_consensus.cc:348] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.302155 14709 raft_consensus.cc:374] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:37.302198 14709 raft_consensus.cc:729] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a381ec4fad64bb5a649c02a4b2992b3, State: Initialized, Role: FOLLOWER
I20260928 02:47:37.302307 14709 consensus_queue.cc:260] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.302593 14709 ts_tablet_manager.cc:1428] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:47:37.302760 14431 heartbeater.cc:503] Master 127.27.109.190:35365 was elected leader, sending a full tablet report...
I20260928 02:47:37.303167 14712 tablet_bootstrap.cc:492] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: No bootstrap required, opened a new log
I20260928 02:47:37.303251 14712 ts_tablet_manager.cc:1397] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:37.305877 14712 raft_consensus.cc:348] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.305989 14712 raft_consensus.cc:374] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:37.306008 14712 raft_consensus.cc:729] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ee947139914386b285f0f3096ca934, State: Initialized, Role: FOLLOWER
I20260928 02:47:37.306133 14712 consensus_queue.cc:260] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.306391 14712 ts_tablet_manager.cc:1428] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:37.306559 14562 heartbeater.cc:503] Master 127.27.109.190:35365 was elected leader, sending a full tablet report...
I20260928 02:47:37.365969 14711 raft_consensus.cc:482] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:37.366082 14711 raft_consensus.cc:504] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.366513 14711 leader_election.cc:288] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a381ec4fad64bb5a649c02a4b2992b3 (127.27.109.129:45815), 65ee947139914386b285f0f3096ca934 (127.27.109.130:44963)
I20260928 02:47:37.369560 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.366785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:47941 (local address 127.27.109.129:45815)
0928 02:47:37.366843 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.366845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.366995 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:47:37.367112 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.367113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.367125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.367216 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:37.367368 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.367901 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.368852 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.369204 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.369290 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.369298 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.369456 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.369490 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:47:37.369590 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.366678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:45815 (local address 127.27.109.131:47941)
0928 02:47:37.366918 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.366927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.367038 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.367201 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.367202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.367204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:37.367287 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.367289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.367987 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.367988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.368757 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.368759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.369290 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.369291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.369370 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.369379 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.369439 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.369478 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":73,"thread_start_us":118,"threads_started":1}
I20260928 02:47:37.369616 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.366737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:41707 (local address 127.27.109.130:44963)
0928 02:47:37.366802 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.366804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.366815 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:37.366892 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.366893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.366903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.367007 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:37.367174 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.367700 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.368802 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.369219 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.369335 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.369342 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.369493 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.369548 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:47:37.369962 14385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bdc63bbf1547408ee2dca337c36a96" candidate_uuid: "c33d4bc069c2453298326a9fb76d8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" is_pre_election: true
I20260928 02:47:37.370113 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.366674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:44963 (local address 127.27.109.131:41707)
0928 02:47:37.366746 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.366758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:37.366804 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:37.367016 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:37.367017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:37.367019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:37.367100 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.367101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.367792 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.367793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.368675 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:37.368677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.369335 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.369336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:37.369406 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.369414 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:47:37.369485 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:47:37.369524 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:47:37.370117 14385 raft_consensus.cc:2454] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4bc069c2453298326a9fb76d8542 in term 0.
I20260928 02:47:37.370433 14583 leader_election.cc:302] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a381ec4fad64bb5a649c02a4b2992b3, c33d4bc069c2453298326a9fb76d8542; no voters:
I20260928 02:47:37.370486 14516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bdc63bbf1547408ee2dca337c36a96" candidate_uuid: "c33d4bc069c2453298326a9fb76d8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ee947139914386b285f0f3096ca934" is_pre_election: true
I20260928 02:47:37.370587 14711 raft_consensus.cc:2781] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:37.370615 14516 raft_consensus.cc:2454] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33d4bc069c2453298326a9fb76d8542 in term 0.
I20260928 02:47:37.370651 14711 raft_consensus.cc:482] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:37.370682 14711 raft_consensus.cc:3037] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:37.371366 14711 raft_consensus.cc:504] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.371500 14711 leader_election.cc:288] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [CANDIDATE]: Term 1 election: Requested vote from peers 8a381ec4fad64bb5a649c02a4b2992b3 (127.27.109.129:45815), 65ee947139914386b285f0f3096ca934 (127.27.109.130:44963)
I20260928 02:47:37.371657 14385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bdc63bbf1547408ee2dca337c36a96" candidate_uuid: "c33d4bc069c2453298326a9fb76d8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a381ec4fad64bb5a649c02a4b2992b3"
I20260928 02:47:37.371680 14516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56bdc63bbf1547408ee2dca337c36a96" candidate_uuid: "c33d4bc069c2453298326a9fb76d8542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65ee947139914386b285f0f3096ca934"
I20260928 02:47:37.371724 14385 raft_consensus.cc:3037] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:37.371783 14516 raft_consensus.cc:3037] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:37.372468 14385 raft_consensus.cc:2454] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4bc069c2453298326a9fb76d8542 in term 1.
I20260928 02:47:37.372475 14516 raft_consensus.cc:2454] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33d4bc069c2453298326a9fb76d8542 in term 1.
I20260928 02:47:37.372618 14583 leader_election.cc:302] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a381ec4fad64bb5a649c02a4b2992b3, c33d4bc069c2453298326a9fb76d8542; no voters:
I20260928 02:47:37.372721 14711 raft_consensus.cc:2781] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:37.372846 14711 raft_consensus.cc:686] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 1 LEADER]: Becoming Leader. State: Replica: c33d4bc069c2453298326a9fb76d8542, State: Running, Role: LEADER
I20260928 02:47:37.372942 14711 consensus_queue.cc:237] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } }
I20260928 02:47:37.373690 14243 catalog_manager.cc:5495] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33d4bc069c2453298326a9fb76d8542 (127.27.109.131). New cstate: current_term: 1 leader_uuid: "c33d4bc069c2453298326a9fb76d8542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33d4bc069c2453298326a9fb76d8542" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 41649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:37.393075 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.390678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.27.109.131:41649)
0928 02:47:37.390723 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:37.390725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:37.390775 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:47:37.390859 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:37.390860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:37.390879 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:37.390970 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:37.391090 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.391536 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.392291 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:37.392488 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:37.392572 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:37.392723 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:37.392814 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:37.392938 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:37.392970 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:47:37.392978 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 02:47:37.394776 14563 tablet.cc:2369] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:37.394758 14385 raft_consensus.cc:1261] T 56bdc63bbf1547408ee2dca337c36a96 P 8a381ec4fad64bb5a649c02a4b2992b3 [term 1 FOLLOWER]: Refusing update from remote peer c33d4bc069c2453298326a9fb76d8542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:37.394989 14516 raft_consensus.cc:1261] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 1 FOLLOWER]: Refusing update from remote peer c33d4bc069c2453298326a9fb76d8542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:37.395172 14717 consensus_queue.cc:1035] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a381ec4fad64bb5a649c02a4b2992b3" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:37.395282 14711 consensus_queue.cc:1035] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ee947139914386b285f0f3096ca934" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:37.397490 14720 mvcc.cc:205] Tried to move back new op lower bound from 7334148740686258176 to 7334148740600225792. Current Snapshot: MvccSnapshot[applied={T|T < 7334148740686258176}]
I20260928 02:47:37.397743 14721 mvcc.cc:205] Tried to move back new op lower bound from 7334148740686258176 to 7334148740600225792. Current Snapshot: MvccSnapshot[applied={T|T < 7334148740686258176}]
I20260928 02:47:37.398715 14722 mvcc.cc:205] Tried to move back new op lower bound from 7334148740686258176 to 7334148740600225792. Current Snapshot: MvccSnapshot[applied={T|T < 7334148740686258176 or (T in {7334148740686258176})}]
I20260928 02:47:37.400017 14242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:37.400110 14242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:37.400854 14242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e1e113eeacd43c4993d13152cdae11d: Sending DeleteTablet for 3 replicas of tablet 56bdc63bbf1547408ee2dca337c36a96
I20260928 02:47:37.401171 14496 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:37 UTC) from {username='slave'} at 127.0.0.1:44286
I20260928 02:47:37.401278 14627 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:37 UTC) from {username='slave'} at 127.0.0.1:34030
I20260928 02:47:37.401307 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14303
I20260928 02:47:37.401379 14365 tablet_service.cc:1514] Processing DeleteTablet for tablet 56bdc63bbf1547408ee2dca337c36a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:37 UTC) from {username='slave'} at 127.0.0.1:50096
I20260928 02:47:37.402604 14729 tablet_replica.cc:330] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: stopping tablet replica
I20260928 02:47:37.402701 14729 raft_consensus.cc:2229] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:37.402787 14729 raft_consensus.cc:2258] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:37.403241 14729 ts_tablet_manager.cc:1905] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:37.404839 14730 tablet_replica.cc:330] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: stopping tablet replica
I20260928 02:47:37.404932 14730 raft_consensus.cc:2229] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:47:37.404958 14729 ts_tablet_manager.cc:1918] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:37.405021 14730 raft_consensus.cc:2258] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:37.405112 14729 log.cc:1196] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/56bdc63bbf1547408ee2dca337c36a96
I20260928 02:47:37.405409 14729 ts_tablet_manager.cc:1939] T 56bdc63bbf1547408ee2dca337c36a96 P 65ee947139914386b285f0f3096ca934: Deleting consensus metadata
I20260928 02:47:37.405442 14730 ts_tablet_manager.cc:1905] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:37.405864 14228 catalog_manager.cc:4841] TS 65ee947139914386b285f0f3096ca934 (127.27.109.130:44963): tablet 56bdc63bbf1547408ee2dca337c36a96 (table test-table [id=ace1dde230194566b94a3a5379b0fbed]) successfully deleted
I20260928 02:47:37.408012 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14434
I20260928 02:47:37.408706 14730 ts_tablet_manager.cc:1918] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:37.408810 14730 log.cc:1196] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/56bdc63bbf1547408ee2dca337c36a96
I20260928 02:47:37.409086 14730 ts_tablet_manager.cc:1939] T 56bdc63bbf1547408ee2dca337c36a96 P c33d4bc069c2453298326a9fb76d8542: Deleting consensus metadata
I20260928 02:47:37.409518 14229 catalog_manager.cc:4841] TS c33d4bc069c2453298326a9fb76d8542 (127.27.109.131:41649): tablet 56bdc63bbf1547408ee2dca337c36a96 (table test-table [id=ace1dde230194566b94a3a5379b0fbed]) successfully deleted
W20260928 02:47:37.410156 14230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:45815 (error 108)
I20260928 02:47:37.414729 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:37.414602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:45815 (local address 127.0.0.1:50100)
0928 02:47:37.414655 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:37.414695 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:45815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:47:37.414860 14230 catalog_manager.cc:4584] TS 8a381ec4fad64bb5a649c02a4b2992b3 (127.27.109.129:45815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56bdc63bbf1547408ee2dca337c36a96: Network error: Client connection negotiation failed: client connection to 127.27.109.129:45815: connect: Connection refused (error 111)
I20260928 02:47:37.415410 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14565
I20260928 02:47:37.421006 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14212
2026-09-28T02:47:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:37 dist-test-slave-b5p5 krb5kdc[14734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:37 dist-test-slave-b5p5 krb5kdc[14734](info): set up 2 sockets
Sep 28 02:47:37 dist-test-slave-b5p5 krb5kdc[14734](info): commencing operation
krb5kdc: starting...
W20260928 02:47:38.442322 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:38 dist-test-slave-b5p5 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:38.552683 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:42333
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:42333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:38.630254 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:38.630472 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:38.630542 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:38.631666 14747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:38.631737 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:38.631764 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:38.631801 14747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:38.631824 14747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:38.633206 14747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:42333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:42333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:38.633462 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:38.633675 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:38.638172 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:38.638230 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:38.638170 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:38.638298 14747 server_base.cc:1034] running on GCE node
I20260928 02:47:38.638612 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:38.638839 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:38.639972 14747 hybrid_clock.cc:648] HybridClock initialized: now 1790563658639956 us; error 27 us; skew 500 ppm
Sep 28 02:47:38 dist-test-slave-b5p5 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563658, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:38.645951 14747 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:38.646399 14747 webserver.cc:460] Webserver started at http://127.27.109.190:33513/ using document root <none> and password file <none>
I20260928 02:47:38.646622 14747 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:38.646665 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:38.646806 14747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:38.647819 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "b0b30c4916ec4def949e77e3baee2851"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "7cfe59f0ce5e4ac8c159dac7bfbb587e"
server_key_iv: "b533db85b2a8af05a6178c8f89700893"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.648170 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "b0b30c4916ec4def949e77e3baee2851"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "7cfe59f0ce5e4ac8c159dac7bfbb587e"
server_key_iv: "b533db85b2a8af05a6178c8f89700893"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.649484 14747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:38.650208 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:38.650388 14747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:38.650527 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "b0b30c4916ec4def949e77e3baee2851"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "7cfe59f0ce5e4ac8c159dac7bfbb587e"
server_key_iv: "b533db85b2a8af05a6178c8f89700893"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.650614 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:38.665899 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:38.667739 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:38.667892 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:38.671653 14747 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:42333
I20260928 02:47:38.671689 14814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:42333 every 8 connection(s)
I20260928 02:47:38.672005 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:38.672808 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:38.675067 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: Bootstrap starting.
I20260928 02:47:38.675623 14815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:38.675910 14815 log.cc:824] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:38.676565 14815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: No bootstrap required, opened a new log
I20260928 02:47:38.676817 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14747
I20260928 02:47:38.676929 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:38.677124 28086 external_mini_cluster.cc:1428] Setting key 56d473dae47460e2eb73f0ed95917254
I20260928 02:47:38.678678 14815 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } }
I20260928 02:47:38.678833 14815 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:38.678872 14815 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b30c4916ec4def949e77e3baee2851, State: Initialized, Role: FOLLOWER
I20260928 02:47:38.679009 14815 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } }
I20260928 02:47:38.679095 14815 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:38.679128 14815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:38.679179 14815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:38.680107 14815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } }
I20260928 02:47:38.680190 14815 leader_election.cc:302] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b30c4916ec4def949e77e3baee2851; no voters:
I20260928 02:47:38.680377 14815 leader_election.cc:288] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:38.680397 14820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:38.680508 14820 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [term 1 LEADER]: Becoming Leader. State: Replica: b0b30c4916ec4def949e77e3baee2851, State: Running, Role: LEADER
I20260928 02:47:38.680588 14820 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } }
I20260928 02:47:38.680624 14815 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:38.680887 14820 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b30c4916ec4def949e77e3baee2851. Latest consensus state: current_term: 1 leader_uuid: "b0b30c4916ec4def949e77e3baee2851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } } }
I20260928 02:47:38.680951 14820 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:38.681499 14825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:38.681998 14825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:38.682029 14821 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b30c4916ec4def949e77e3baee2851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b30c4916ec4def949e77e3baee2851" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42333 } } }
I20260928 02:47:38.682283 14821 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:38.683872 14825 catalog_manager.cc:1347] Generated new cluster ID: 814fd83c1424446e837783c6eb2f56ce
I20260928 02:47:38.683923 14825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:47:38 dist-test-slave-b5p5 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:38.687960 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:38.677587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.27.109.190:42333)
0928 02:47:38.677759 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:38.677767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:38.677793 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:47:38.678663 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:38.678669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:38.678714 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:38.678823 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:38.679617 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.680671 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.681770 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.682401 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.682560 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:38.684923 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:38.684927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:38.684927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:38.684938 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:38.686413 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:38.686811 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:38.686814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:38.686814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:38.686928 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:38.687139 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:38.687140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:38.687141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:38.687376 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:38.687543 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:38.687631 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:47:38.687766 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260928 02:47:38.696702 14825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:38.697171 14825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:38.701356 14825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: Generated new TSK 0
I20260928 02:47:38.701514 14825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
Entry for principal kudu/127.27.109.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab.
I20260928 02:47:38.731436 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:38.829478 14841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:38.829661 14841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:38.829684 14841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:38.830814 14841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:38.830852 14841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:38.830878 14841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:38.832263 14841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.129.keytab
--principal=kudu/127.27.109.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:38.832481 14841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:38.832713 14841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:38.838231 14847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:38.838258 14846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:38.838285 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:38.838433 14841 server_base.cc:1034] running on GCE node
I20260928 02:47:38.838773 14841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:38.839002 14841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:38.840157 14841 hybrid_clock.cc:648] HybridClock initialized: now 1790563658840143 us; error 33 us; skew 500 ppm
Sep 28 02:47:38 dist-test-slave-b5p5 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563658, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:38.842454 14841 init.cc:377] Logged in from keytab as kudu/127.27.109.129@KRBTEST.COM (short username kudu)
I20260928 02:47:38.842952 14841 webserver.cc:460] Webserver started at http://127.27.109.129:33871/ using document root <none> and password file <none>
I20260928 02:47:38.843120 14841 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:38.843158 14841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:38.843286 14841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:38.844213 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "5cd774ad07274e50af837591198c7f35"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "24b17f8cf60e91e63ff5250db441234e"
server_key_iv: "33b1bb58dd74116ce449295825ba1974"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.844502 14841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "5cd774ad07274e50af837591198c7f35"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "24b17f8cf60e91e63ff5250db441234e"
server_key_iv: "33b1bb58dd74116ce449295825ba1974"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.845682 14841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:38.846347 14856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:38.846526 14841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:38.846611 14841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "5cd774ad07274e50af837591198c7f35"
format_stamp: "Formatted at 2026-09-28 02:47:38 on dist-test-slave-b5p5"
server_key: "24b17f8cf60e91e63ff5250db441234e"
server_key_iv: "33b1bb58dd74116ce449295825ba1974"
server_key_version: "encryptionkey@0"
I20260928 02:47:38.846664 14841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:38.877535 14841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:38.879760 14841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:38.879869 14841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:38.880044 14841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:38.880366 14841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:38.880402 14841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:38.880447 14841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:38.880478 14841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:38.886124 14841 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:34281
I20260928 02:47:38.886265 14969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:34281 every 8 connection(s)
I20260928 02:47:38.886544 14841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:47:38 dist-test-slave-b5p5 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563658, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.129@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:38.895248 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:38.887261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:54525 (local address 127.27.109.190:42333)
0928 02:47:38.887313 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:38.887316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:38.887495 (+ 179us) server_negotiation.cc:407] Connection header received
0928 02:47:38.888027 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:38.888028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:38.888045 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:38.888091 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:38.889021 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.889526 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.890518 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.890733 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.890808 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:38.892727 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:38.892729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:38.892729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:38.892739 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:38.893915 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:38.894312 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:38.894313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:38.894314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:38.894371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:38.894637 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:38.894638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:38.894638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:38.894745 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:38.894822 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:38.895076 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:47:38.895159 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:47:38.895421 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:38.887219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:42333 (local address 127.27.109.129:54525)
0928 02:47:38.887358 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:38.887413 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:38.887911 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:38.888172 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:38.888175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:38.888731 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:38.888920 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:38.888925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.889621 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.889622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:38.890404 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:38.890406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:38.890829 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:38.890830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:38.890947 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:38.891354 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:38.891362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:38.892630 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:38.894011 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:38.894018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:38.894020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:38.894227 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:38.894438 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:38.894439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:38.894439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:38.894567 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:38.894814 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:38.894821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:38.894949 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:47:38.895055 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:47:38.895198 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 02:47:38.895921 14970 heartbeater.cc:347] Connected to a master server at 127.27.109.190:42333
I20260928 02:47:38.896004 14970 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:38.896127 14970 heartbeater.cc:511] Master 127.27.109.190:42333 requested a full tablet report, sending...
I20260928 02:47:38.896407 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14841
I20260928 02:47:38.896538 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:38.896692 14779 ts_manager.cc:194] Registered new tserver with Master: 5cd774ad07274e50af837591198c7f35 (127.27.109.129:34281)
I20260928 02:47:38.896742 28086 external_mini_cluster.cc:1428] Setting key 0e9b55a6dc24bbcc15df0f279e6b0964
I20260928 02:47:38.897554 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.129@KRBTEST.COM'} at 127.27.109.129:54525
WARNING: no policy specified for kudu/127.27.109.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
Entry for principal kudu/127.27.109.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab.
I20260928 02:47:38.919445 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:38.999105 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:38.999286 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:38.999308 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.000415 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.000453 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.000479 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:39.001850 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.130.keytab
--principal=kudu/127.27.109.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.002067 14975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.002276 14975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:39.007570 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.007629 14975 server_base.cc:1034] running on GCE node
W20260928 02:47:39.007601 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.007570 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.007905 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:39.008133 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:39.009286 14975 hybrid_clock.cc:648] HybridClock initialized: now 1790563659009275 us; error 25 us; skew 500 ppm
Sep 28 02:47:39 dist-test-slave-b5p5 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563659, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:39.011731 14975 init.cc:377] Logged in from keytab as kudu/127.27.109.130@KRBTEST.COM (short username kudu)
I20260928 02:47:39.012167 14975 webserver.cc:460] Webserver started at http://127.27.109.130:42661/ using document root <none> and password file <none>
I20260928 02:47:39.012343 14975 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:39.012384 14975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:39.012466 14975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:39.013435 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "3079af78d46e3f3b7f5d0d30b486ace2"
server_key_iv: "5290964a86751136e2616a14c3520b12"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.013715 14975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "3079af78d46e3f3b7f5d0d30b486ace2"
server_key_iv: "5290964a86751136e2616a14c3520b12"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.015044 14975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:39.015753 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.015940 14975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:39.016021 14975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "3079af78d46e3f3b7f5d0d30b486ace2"
server_key_iv: "5290964a86751136e2616a14c3520b12"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.016072 14975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:39.035063 14975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:39.036773 14975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:39.036921 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:39.037124 14975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:39.037416 14975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:39.037448 14975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.037468 14975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:39.037515 14975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.042896 14975 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:40417
I20260928 02:47:39.042953 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:40417 every 8 connection(s)
I20260928 02:47:39.043262 14975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:39.043499 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 14975
I20260928 02:47:39.043630 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:39.043841 28086 external_mini_cluster.cc:1428] Setting key 1a538552fe4415115577271a9eac86c8
Sep 28 02:47:39 dist-test-slave-b5p5 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563659, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.130@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.131@KRBTEST.COM; defaulting to no policy
I20260928 02:47:39.053901 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.044086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:46333 (local address 127.27.109.190:42333)
0928 02:47:39.044201 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.044203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.044372 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:47:39.044925 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.044926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.044937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.044993 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.046080 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.046668 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.048206 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.048398 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.048455 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.050875 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.050876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.050876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:39.050883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.052448 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.052901 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.052902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.052902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.052959 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.053255 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.053256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.053256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.053376 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.053451 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.053718 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.053796 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:47:39.054136 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.044085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:42333 (local address 127.27.109.130:46333)
0928 02:47:39.044259 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.044300 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.044821 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.045058 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.045062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.045700 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:39.045968 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.045974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.046786 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.046787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.048117 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.048121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.048485 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.048486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.048646 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.049115 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:39.049125 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:39.050764 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:39.052530 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:39.052536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:39.052539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:39.052816 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:39.053042 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:39.053043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:39.053043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:39.053143 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:39.053455 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:39.053461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:39.053581 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.053696 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.053926 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.131@KRBTEST.COM" created.
I20260928 02:47:39.054605 15104 heartbeater.cc:347] Connected to a master server at 127.27.109.190:42333
I20260928 02:47:39.054715 15104 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:39.054883 15104 heartbeater.cc:511] Master 127.27.109.190:42333 requested a full tablet report, sending...
I20260928 02:47:39.055269 14779 ts_manager.cc:194] Registered new tserver with Master: 1a03d1b784fc4ec39c3da1409f1a4bc9 (127.27.109.130:40417)
I20260928 02:47:39.055627 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.130@KRBTEST.COM'} at 127.27.109.130:46333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
Entry for principal kudu/127.27.109.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab.
I20260928 02:47:39.067329 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:39.146697 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:39.146898 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:39.146924 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.148046 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.148084 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.148110 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:39.149600 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.131.keytab
--principal=kudu/127.27.109.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.149843 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.150063 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:39.154937 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.154943 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.155040 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.155242 15109 server_base.cc:1034] running on GCE node
I20260928 02:47:39.155422 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:39.155654 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:39.156821 15109 hybrid_clock.cc:648] HybridClock initialized: now 1790563659156817 us; error 36 us; skew 500 ppm
Sep 28 02:47:39 dist-test-slave-b5p5 krb5kdc[14734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563659, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:39.159322 15109 init.cc:377] Logged in from keytab as kudu/127.27.109.131@KRBTEST.COM (short username kudu)
I20260928 02:47:39.159729 15109 webserver.cc:460] Webserver started at http://127.27.109.131:42137/ using document root <none> and password file <none>
I20260928 02:47:39.159941 15109 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:39.160007 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:39.160154 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:39.161201 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "6384e54229164f578d57218c9ce0129c"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "6fe9d5699a576df3417f76139065c384"
server_key_iv: "bf9eebeba204d1f1fa1c30fe27a573a8"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.161554 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "6384e54229164f578d57218c9ce0129c"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "6fe9d5699a576df3417f76139065c384"
server_key_iv: "bf9eebeba204d1f1fa1c30fe27a573a8"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.162863 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:47:39.163532 15124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.163688 15109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:39.163781 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "6384e54229164f578d57218c9ce0129c"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "6fe9d5699a576df3417f76139065c384"
server_key_iv: "bf9eebeba204d1f1fa1c30fe27a573a8"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.163859 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:39.185392 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:39.187636 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:39.187800 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:39.188028 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:39.188310 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:39.188362 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.188402 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:39.188431 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.194016 15109 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:44381
I20260928 02:47:39.194103 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:44381 every 8 connection(s)
I20260928 02:47:39.194389 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:47:39 dist-test-slave-b5p5 krb5kdc[14734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563659, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.131@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:39.201478 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15109
I20260928 02:47:39.201573 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:39.201798 28086 external_mini_cluster.cc:1428] Setting key 45c3ff43b07d47d96b555c39ba4fe9ae
I20260928 02:47:39.203210 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.195090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:51107 (local address 127.27.109.190:42333)
0928 02:47:39.195163 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.195165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.195335 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:47:39.195785 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.195786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.195796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.195856 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.196682 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.197156 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.198073 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.198256 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.198306 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.200306 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.200307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.200307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:39.200313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.201908 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.202344 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.202345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.202346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.202404 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.202693 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.202694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.202694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.202826 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.202895 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.203095 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.203151 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:47:39.203444 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.195048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:42333 (local address 127.27.109.131:51107)
0928 02:47:39.195211 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.195251 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.195701 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.195907 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.195909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.196433 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:39.196608 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.196612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.197236 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.197237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.198001 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.198003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.198326 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.198327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.198419 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.198904 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:47:39.198912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:39.200227 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:39.201987 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:39.201994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:39.201996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:39.202273 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:39.202495 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:47:39.202495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:47:39.202496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:47:39.202624 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:47:39.202893 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:39.202898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:39.203002 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.203081 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.203201 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 02:47:39.203876 15238 heartbeater.cc:347] Connected to a master server at 127.27.109.190:42333
I20260928 02:47:39.203948 15238 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:39.204077 15238 heartbeater.cc:511] Master 127.27.109.190:42333 requested a full tablet report, sending...
I20260928 02:47:39.204380 14779 ts_manager.cc:194] Registered new tserver with Master: 6384e54229164f578d57218c9ce0129c (127.27.109.131:44381)
I20260928 02:47:39.204785 14779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.131@KRBTEST.COM'} at 127.27.109.131:51107
I20260928 02:47:39.213475 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:39.220362 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.27.109.190:42333)
0928 02:47:39.214848 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.214850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.214895 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:47:39.214944 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.214944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.214951 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.214998 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.215505 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.215961 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.216778 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.216950 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.217004 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.218034 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.218035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.218036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:39.218041 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.219444 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.219792 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.219793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.219793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.219841 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:39.219992 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:39.219993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:39.219993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:39.220105 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.220171 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.220255 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.220306 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:47:39.221969 14779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:39.223569 14779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:39.229753 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.225677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:40417 (local address 127.0.0.1:51560)
0928 02:47:39.225926 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.225937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.225988 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.226411 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.226412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.226413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:39.226450 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.226451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227813 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.227814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.228622 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.228624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229490 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.229490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.229550 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.229554 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.229591 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.229649 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260928 02:47:39.229795 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.225947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.27.109.129:34281)
0928 02:47:39.226103 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.226108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.226193 (+ 85us) server_negotiation.cc:407] Connection header received
0928 02:47:39.226240 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.226242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.226257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.226349 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:39.226514 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227332 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.228796 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229336 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.229441 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.229496 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.229664 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.229709 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260928 02:47:39.229941 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.225677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:44381 (local address 127.0.0.1:37210)
0928 02:47:39.225835 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.225870 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.225935 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.226389 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.226390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.226393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:39.226446 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.226448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227772 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.227774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.228878 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.228880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229694 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.229695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.229765 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.229769 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.229800 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.229839 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":31,"thread_start_us":59,"threads_started":1}
I20260928 02:47:39.229851 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.225699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34281 (local address 127.0.0.1:43332)
0928 02:47:39.226112 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.226121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.226175 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.226356 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.226360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.226362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:39.226425 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.226427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227445 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.227446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.228682 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.228685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229446 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.229446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.229535 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.229548 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.229648 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.229692 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":80,"threads_started":1}
I20260928 02:47:39.230124 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.226036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.27.109.131:44381)
0928 02:47:39.226201 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.226205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.226216 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:39.226262 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.226264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.226279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.226544 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:39.226605 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227661 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.228955 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229606 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.229900 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.229961 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.229984 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.230022 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260928 02:47:39.230422 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.225877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.27.109.130:40417)
0928 02:47:39.226233 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.226237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.226245 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:47:39.226280 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.226282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.226296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.226585 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:39.226649 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.227687 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.228726 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.229389 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.230277 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.230317 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.230337 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.230367 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":78,"threads_started":1}
I20260928 02:47:39.230794 15172 tablet_service.cc:1467] Processing CreateTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 (DEFAULT_TABLE table=test-table [id=0cc02f8c5db8495494b0afcf5cbc5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:39.230824 14904 tablet_service.cc:1467] Processing CreateTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 (DEFAULT_TABLE table=test-table [id=0cc02f8c5db8495494b0afcf5cbc5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:39.231123 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f971ba82a9c40b2bf7c002f2933a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:39.231123 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f971ba82a9c40b2bf7c002f2933a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:39.231102 15038 tablet_service.cc:1467] Processing CreateTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 (DEFAULT_TABLE table=test-table [id=0cc02f8c5db8495494b0afcf5cbc5705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:39.231352 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f971ba82a9c40b2bf7c002f2933a572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:39.233942 15255 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Bootstrap starting.
I20260928 02:47:39.234244 15256 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Bootstrap starting.
I20260928 02:47:39.234453 15257 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Bootstrap starting.
I20260928 02:47:39.234642 15255 tablet_bootstrap.cc:654] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:39.235143 15257 tablet_bootstrap.cc:654] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:39.235230 15255 log.cc:824] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:39.235678 15257 log.cc:824] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:39.236074 15256 tablet_bootstrap.cc:654] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:39.236449 15257 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: No bootstrap required, opened a new log
I20260928 02:47:39.236529 15257 ts_tablet_manager.cc:1397] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:39.236653 15256 log.cc:824] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:39.237175 15255 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: No bootstrap required, opened a new log
I20260928 02:47:39.237251 15255 ts_tablet_manager.cc:1397] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:47:39.237468 15256 tablet_bootstrap.cc:492] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: No bootstrap required, opened a new log
I20260928 02:47:39.237561 15256 ts_tablet_manager.cc:1397] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:47:39.238579 15257 raft_consensus.cc:348] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.238768 15257 raft_consensus.cc:374] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:39.238804 15257 raft_consensus.cc:729] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd774ad07274e50af837591198c7f35, State: Initialized, Role: FOLLOWER
I20260928 02:47:39.238936 15257 consensus_queue.cc:260] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.239041 15256 raft_consensus.cc:348] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.239217 15257 ts_tablet_manager.cc:1428] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:39.239236 15256 raft_consensus.cc:374] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:39.239346 14970 heartbeater.cc:503] Master 127.27.109.190:42333 was elected leader, sending a full tablet report...
I20260928 02:47:39.239282 15255 raft_consensus.cc:348] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.239425 15255 raft_consensus.cc:374] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:39.239488 15255 raft_consensus.cc:729] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a03d1b784fc4ec39c3da1409f1a4bc9, State: Initialized, Role: FOLLOWER
I20260928 02:47:39.239594 15255 consensus_queue.cc:260] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.239277 15256 raft_consensus.cc:729] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6384e54229164f578d57218c9ce0129c, State: Initialized, Role: FOLLOWER
I20260928 02:47:39.239776 15256 consensus_queue.cc:260] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.239876 15255 ts_tablet_manager.cc:1428] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:47:39.239938 15104 heartbeater.cc:503] Master 127.27.109.190:42333 was elected leader, sending a full tablet report...
I20260928 02:47:39.240058 15256 ts_tablet_manager.cc:1428] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:47:39.240118 15238 heartbeater.cc:503] Master 127.27.109.190:42333 was elected leader, sending a full tablet report...
W20260928 02:47:39.293850 15105 tablet.cc:2369] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:47:39.387286 14971 tablet.cc:2369] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:39.412201 15262 raft_consensus.cc:482] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:39.412338 15262 raft_consensus.cc:504] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.412806 15262 leader_election.cc:288] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6384e54229164f578d57218c9ce0129c (127.27.109.131:44381), 5cd774ad07274e50af837591198c7f35 (127.27.109.129:34281)
I20260928 02:47:39.416668 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.413277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:32863 (local address 127.27.109.131:44381)
0928 02:47:39.413329 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.413331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.413340 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:47:39.413367 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.413368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.413377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.413443 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:39.413643 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.414524 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.415710 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.416308 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.416381 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.416391 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.416564 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.416604 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:47:39.416668 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.412961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:44381 (local address 127.27.109.130:32863)
0928 02:47:39.413082 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.413093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.413135 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.413479 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.413480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.413482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:39.413552 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.413553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.414621 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.414623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.415595 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.415597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.416388 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.416389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.416470 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.416479 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.416555 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.416598 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 02:47:39.416970 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.413040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:55785 (local address 127.27.109.129:34281)
0928 02:47:39.413157 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.413160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.413443 (+ 283us) server_negotiation.cc:407] Connection header received
0928 02:47:39.413651 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.413652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.413661 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.413729 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:39.413915 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.414634 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.415999 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.416573 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.416647 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.416655 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.416872 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.416909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:47:39.416983 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.412963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34281 (local address 127.27.109.130:55785)
0928 02:47:39.413373 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.413383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.413440 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.413757 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.413758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.413760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:39.413825 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.413827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.414754 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.414755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.415893 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.415896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.416703 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.416705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.416788 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.416793 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.416849 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.416890 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":220,"thread_start_us":129,"threads_started":1}
I20260928 02:47:39.417234 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f971ba82a9c40b2bf7c002f2933a572" candidate_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6384e54229164f578d57218c9ce0129c" is_pre_election: true
I20260928 02:47:39.417351 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f971ba82a9c40b2bf7c002f2933a572" candidate_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd774ad07274e50af837591198c7f35" is_pre_election: true
I20260928 02:47:39.417402 15192 raft_consensus.cc:2454] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a03d1b784fc4ec39c3da1409f1a4bc9 in term 0.
I20260928 02:47:39.417480 14924 raft_consensus.cc:2454] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a03d1b784fc4ec39c3da1409f1a4bc9 in term 0.
I20260928 02:47:39.417642 14992 leader_election.cc:302] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a03d1b784fc4ec39c3da1409f1a4bc9, 6384e54229164f578d57218c9ce0129c; no voters:
I20260928 02:47:39.417793 15262 raft_consensus.cc:2781] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:39.417856 15262 raft_consensus.cc:482] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:39.417881 15262 raft_consensus.cc:3037] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:39.418720 15262 raft_consensus.cc:504] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.418855 15262 leader_election.cc:288] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 6384e54229164f578d57218c9ce0129c (127.27.109.131:44381), 5cd774ad07274e50af837591198c7f35 (127.27.109.129:34281)
I20260928 02:47:39.418982 15192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f971ba82a9c40b2bf7c002f2933a572" candidate_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6384e54229164f578d57218c9ce0129c"
I20260928 02:47:39.419054 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f971ba82a9c40b2bf7c002f2933a572" candidate_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd774ad07274e50af837591198c7f35"
I20260928 02:47:39.419057 15192 raft_consensus.cc:3037] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:39.419114 14924 raft_consensus.cc:3037] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:39.419953 14924 raft_consensus.cc:2454] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a03d1b784fc4ec39c3da1409f1a4bc9 in term 1.
I20260928 02:47:39.419953 15192 raft_consensus.cc:2454] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a03d1b784fc4ec39c3da1409f1a4bc9 in term 1.
I20260928 02:47:39.420130 14992 leader_election.cc:302] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a03d1b784fc4ec39c3da1409f1a4bc9, 6384e54229164f578d57218c9ce0129c; no voters:
I20260928 02:47:39.420258 15262 raft_consensus.cc:2781] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:39.420420 15262 raft_consensus.cc:686] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 1a03d1b784fc4ec39c3da1409f1a4bc9, State: Running, Role: LEADER
I20260928 02:47:39.420545 15262 consensus_queue.cc:237] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } }
I20260928 02:47:39.421341 14779 catalog_manager.cc:5495] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a03d1b784fc4ec39c3da1409f1a4bc9 (127.27.109.130). New cstate: current_term: 1 leader_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a03d1b784fc4ec39c3da1409f1a4bc9" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 40417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 } health_report { overall_health: UNKNOWN } } }
W20260928 02:47:39.444938 15239 tablet.cc:2369] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:47:39.455698 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.453510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.27.109.130:40417)
0928 02:47:39.453567 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.453570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.453584 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:47:39.453614 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.453615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.453630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.453692 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:39.453804 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.454247 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.455013 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.455188 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.455238 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.455361 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:39.455457 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:39.455517 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.455612 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.455628 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:47:39.457674 15192 raft_consensus.cc:1261] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 1 FOLLOWER]: Refusing update from remote peer 1a03d1b784fc4ec39c3da1409f1a4bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:39.457695 14924 raft_consensus.cc:1261] T 0f971ba82a9c40b2bf7c002f2933a572 P 5cd774ad07274e50af837591198c7f35 [term 1 FOLLOWER]: Refusing update from remote peer 1a03d1b784fc4ec39c3da1409f1a4bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:39.458024 15265 consensus_queue.cc:1035] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cd774ad07274e50af837591198c7f35" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:39.458137 15262 consensus_queue.cc:1035] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6384e54229164f578d57218c9ce0129c" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:39.460855 15269 mvcc.cc:205] Tried to move back new op lower bound from 7334148749135503360 to 7334148748987129856. Current Snapshot: MvccSnapshot[applied={T|T < 7334148749135503360 or (T in {7334148749135503360})}]
I20260928 02:47:39.462986 14779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:39.463085 14779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:39.463821 14779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0b30c4916ec4def949e77e3baee2851: Sending DeleteTablet for 3 replicas of tablet 0f971ba82a9c40b2bf7c002f2933a572
I20260928 02:47:39.464178 14904 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:43332
I20260928 02:47:39.464175 15038 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:51560
I20260928 02:47:39.464483 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14841
I20260928 02:47:39.464586 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f971ba82a9c40b2bf7c002f2933a572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:39 UTC) from {username='kudu', principal='kudu/127.27.109.190@KRBTEST.COM'} at 127.0.0.1:37210
I20260928 02:47:39.465068 15280 tablet_replica.cc:330] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: stopping tablet replica
I20260928 02:47:39.465171 15280 raft_consensus.cc:2229] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:39.465276 15280 raft_consensus.cc:2258] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:39.465440 15278 tablet_replica.cc:330] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: stopping tablet replica
I20260928 02:47:39.465528 15278 raft_consensus.cc:2229] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:47:39.465636 15278 raft_consensus.cc:2258] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:39.466060 15280 ts_tablet_manager.cc:1905] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:39.466266 15278 ts_tablet_manager.cc:1905] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:39.468139 15280 ts_tablet_manager.cc:1918] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:39.468218 15280 log.cc:1196] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/0f971ba82a9c40b2bf7c002f2933a572
I20260928 02:47:39.468567 15280 ts_tablet_manager.cc:1939] T 0f971ba82a9c40b2bf7c002f2933a572 P 6384e54229164f578d57218c9ce0129c: Deleting consensus metadata
I20260928 02:47:39.468178 15278 ts_tablet_manager.cc:1918] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:39.468761 15278 log.cc:1196] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/0f971ba82a9c40b2bf7c002f2933a572
I20260928 02:47:39.468968 14764 catalog_manager.cc:4841] TS 6384e54229164f578d57218c9ce0129c (127.27.109.131:44381): tablet 0f971ba82a9c40b2bf7c002f2933a572 (table test-table [id=0cc02f8c5db8495494b0afcf5cbc5705]) successfully deleted
I20260928 02:47:39.469132 15278 ts_tablet_manager.cc:1939] T 0f971ba82a9c40b2bf7c002f2933a572 P 1a03d1b784fc4ec39c3da1409f1a4bc9: Deleting consensus metadata
I20260928 02:47:39.469547 14765 catalog_manager.cc:4841] TS 1a03d1b784fc4ec39c3da1409f1a4bc9 (127.27.109.130:40417): tablet 0f971ba82a9c40b2bf7c002f2933a572 (table test-table [id=0cc02f8c5db8495494b0afcf5cbc5705]) successfully deleted
W20260928 02:47:39.471975 14763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:34281 (error 108)
I20260928 02:47:39.472214 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14975
I20260928 02:47:39.472970 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.472864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:34281 (local address 127.0.0.1:43348)
0928 02:47:39.472908 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.472940 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:34281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:47:39.473076 14763 catalog_manager.cc:4584] TS 5cd774ad07274e50af837591198c7f35 (127.27.109.129:34281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f971ba82a9c40b2bf7c002f2933a572: Network error: Client connection negotiation failed: client connection to 127.27.109.129:34281: connect: Connection refused (error 111)
I20260928 02:47:39.476337 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15109
I20260928 02:47:39.480379 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 14747
2026-09-28T02:47:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2059 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:39Z Disabled control of system clock
I20260928 02:47:39.506942 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:43909
--webserver_interface=127.27.109.190
--webserver_port=0
--builtin_ntp_servers=127.27.109.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:39.585544 15285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:39.585738 15285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:39.585758 15285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.586824 15285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:39.586858 15285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.586871 15285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.586884 15285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:39.586895 15285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:39.588110 15285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:43909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.588279 15285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.588476 15285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:39.593067 15290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.593093 15291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.593149 15285 server_base.cc:1034] running on GCE node
W20260928 02:47:39.593103 15293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.593484 15285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:39.593747 15285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:39.594914 15285 hybrid_clock.cc:648] HybridClock initialized: now 1790563659594909 us; error 38 us; skew 500 ppm
I20260928 02:47:39.596040 15285 webserver.cc:460] Webserver started at http://127.27.109.190:36011/ using document root <none> and password file <none>
I20260928 02:47:39.596299 15285 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:39.596369 15285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:39.596514 15285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:39.597470 15285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "c999cbe95af3418b88869981fca0d084"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "ad2c523064a17bf7ec44d6976a9e2055"
server_key_iv: "200e5f2056d76ebd493b79170ed74e73"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.597819 15285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "c999cbe95af3418b88869981fca0d084"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "ad2c523064a17bf7ec44d6976a9e2055"
server_key_iv: "200e5f2056d76ebd493b79170ed74e73"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.599129 15285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:39.599833 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.599992 15285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:39.600090 15285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "c999cbe95af3418b88869981fca0d084"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "ad2c523064a17bf7ec44d6976a9e2055"
server_key_iv: "200e5f2056d76ebd493b79170ed74e73"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.600217 15285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:39.612967 15285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:39.613232 15285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:39.613379 15285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:39.616967 15285 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:43909
I20260928 02:47:39.617023 15351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:43909 every 8 connection(s)
I20260928 02:47:39.617362 15285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:39.618198 15352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:39.620272 15352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: Bootstrap starting.
I20260928 02:47:39.620571 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15285
I20260928 02:47:39.620695 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:39.620882 28086 external_mini_cluster.cc:1428] Setting key 8706781a4e8b51ddc66efcbd40b40a7f
I20260928 02:47:39.620886 15352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:39.621166 15352 log.cc:824] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:39.622028 15352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: No bootstrap required, opened a new log
I20260928 02:47:39.624118 15352 raft_consensus.cc:348] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } }
I20260928 02:47:39.624272 15352 raft_consensus.cc:374] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:39.624313 15352 raft_consensus.cc:729] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c999cbe95af3418b88869981fca0d084, State: Initialized, Role: FOLLOWER
I20260928 02:47:39.624454 15352 consensus_queue.cc:260] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } }
I20260928 02:47:39.624529 15352 raft_consensus.cc:388] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:39.624580 15352 raft_consensus.cc:482] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:39.624620 15352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:39.625237 15352 raft_consensus.cc:504] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } }
I20260928 02:47:39.625368 15352 leader_election.cc:302] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c999cbe95af3418b88869981fca0d084; no voters:
I20260928 02:47:39.625537 15352 leader_election.cc:288] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:39.625614 15357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:39.625794 15352 sys_catalog.cc:564] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:39.625900 15357 raft_consensus.cc:686] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [term 1 LEADER]: Becoming Leader. State: Replica: c999cbe95af3418b88869981fca0d084, State: Running, Role: LEADER
I20260928 02:47:39.626019 15357 consensus_queue.cc:237] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } }
I20260928 02:47:39.626511 15358 sys_catalog.cc:455] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c999cbe95af3418b88869981fca0d084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } } }
I20260928 02:47:39.626600 15358 sys_catalog.cc:458] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:39.626753 15358 sys_catalog.cc:455] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c999cbe95af3418b88869981fca0d084. Latest consensus state: current_term: 1 leader_uuid: "c999cbe95af3418b88869981fca0d084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c999cbe95af3418b88869981fca0d084" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 43909 } } }
I20260928 02:47:39.626801 15358 sys_catalog.cc:458] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:39.626852 15366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:39.627197 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.621333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.27.109.190:43909)
0928 02:47:39.621509 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.621516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.621539 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:47:39.621932 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.621939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.621990 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.622129 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.622362 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.623287 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.624438 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.625059 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.625184 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.625831 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.625835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.625835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:39.626044 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.626081 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:39.626094 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.626152 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.626288 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.626376 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260928 02:47:39.627329 15366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:39.628831 15366 catalog_manager.cc:1347] Generated new cluster ID: ae5d2ab4364b41c7bd1cbbb0f0606df7
I20260928 02:47:39.628887 15366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:39.643059 15366 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:39.643712 15366 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:39.652205 15366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: Generated new TSK 0
I20260928 02:47:39.652359 15366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:39.661221 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.129:0
--local_ip_for_outbound_sockets=127.27.109.129
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--builtin_ntp_servers=127.27.109.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:47:39.739812 15376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:39.740015 15376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:39.740048 15376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.741138 15376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.741197 15376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.741237 15376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.129
I20260928 02:47:39.742563 15376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.129
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.129
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.742795 15376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.743073 15376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:39.748675 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.748612 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.748620 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.748888 15376 server_base.cc:1034] running on GCE node
I20260928 02:47:39.749058 15376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:39.749315 15376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:39.750509 15376 hybrid_clock.cc:648] HybridClock initialized: now 1790563659750447 us; error 33 us; skew 500 ppm
I20260928 02:47:39.751819 15376 webserver.cc:460] Webserver started at http://127.27.109.129:40299/ using document root <none> and password file <none>
I20260928 02:47:39.752076 15376 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:39.752120 15376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:39.752257 15376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:39.753260 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data/instance:
uuid: "4abad56d4cac483783dc4f6fad695e73"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "585c88e8831f240842f736dd1d2a3722"
server_key_iv: "8361e43ca8e12028bb206d2133dc8895"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.753602 15376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance:
uuid: "4abad56d4cac483783dc4f6fad695e73"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "585c88e8831f240842f736dd1d2a3722"
server_key_iv: "8361e43ca8e12028bb206d2133dc8895"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.755002 15376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:39.755775 15390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.755939 15376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:39.756026 15376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal
uuid: "4abad56d4cac483783dc4f6fad695e73"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "585c88e8831f240842f736dd1d2a3722"
server_key_iv: "8361e43ca8e12028bb206d2133dc8895"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.756078 15376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:39.768811 15376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:39.769017 15376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:39.769109 15376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:39.769275 15376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:39.769505 15376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:39.769532 15376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.769552 15376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:39.769564 15376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.774802 15376 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.129:46655
I20260928 02:47:39.774868 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.129:46655 every 8 connection(s)
I20260928 02:47:39.775157 15376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/data/info.pb
I20260928 02:47:39.775593 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15376
I20260928 02:47:39.775712 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-0/wal/instance
I20260928 02:47:39.775895 28086 external_mini_cluster.cc:1428] Setting key 7276a2c2a9350e2268dd1cf737001d08
I20260928 02:47:39.776795 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.130:0
--local_ip_for_outbound_sockets=127.27.109.130
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--builtin_ntp_servers=127.27.109.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:47:39.780295 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.775786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:52763 (local address 127.27.109.190:43909)
0928 02:47:39.775842 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.775844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.776130 (+ 286us) server_negotiation.cc:407] Connection header received
0928 02:47:39.776616 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.776617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.776630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.776676 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.777378 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.777874 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.778920 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.779133 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.779210 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.780046 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.780047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.780047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:39.780051 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.780064 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:39.780068 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.780111 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.780223 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.780240 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:47:39.780589 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.775796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:43909 (local address 127.27.109.129:52763)
0928 02:47:39.775999 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.776036 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.776513 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.776737 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.776741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.776751 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:39.777285 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.777290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.777970 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.777971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.778834 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.778837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.779217 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.779218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.779354 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.779882 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:39.779883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:39.779923 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:39.779952 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:39.780134 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:39.780141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:39.780144 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.780219 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.780338 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260928 02:47:39.781128 15504 heartbeater.cc:347] Connected to a master server at 127.27.109.190:43909
I20260928 02:47:39.781208 15504 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:39.781383 15504 heartbeater.cc:511] Master 127.27.109.190:43909 requested a full tablet report, sending...
I20260928 02:47:39.781847 15316 ts_manager.cc:194] Registered new tserver with Master: 4abad56d4cac483783dc4f6fad695e73 (127.27.109.129:46655)
I20260928 02:47:39.782424 15316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.129:52763
W20260928 02:47:39.855170 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:39.855386 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:39.855417 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.856498 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.856559 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.856596 15507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.130
I20260928 02:47:39.857890 15507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.130
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.130
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.858135 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.858448 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:39.863129 15513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.863139 15515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:39.863139 15512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:39.863480 15507 server_base.cc:1034] running on GCE node
I20260928 02:47:39.863636 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:39.863816 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:39.864980 15507 hybrid_clock.cc:648] HybridClock initialized: now 1790563659864967 us; error 30 us; skew 500 ppm
I20260928 02:47:39.866109 15507 webserver.cc:460] Webserver started at http://127.27.109.130:37505/ using document root <none> and password file <none>
I20260928 02:47:39.866344 15507 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:39.866410 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:39.866559 15507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:39.867568 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data/instance:
uuid: "84de40f7bd6d46e8afd909924d5b7083"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "61bdcb74adb7292299949f40f1a452b6"
server_key_iv: "2408742ea0f9859be55b60a14dda0553"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.867910 15507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance:
uuid: "84de40f7bd6d46e8afd909924d5b7083"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "61bdcb74adb7292299949f40f1a452b6"
server_key_iv: "2408742ea0f9859be55b60a14dda0553"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.869163 15507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:39.869849 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.870016 15507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:39.870110 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal
uuid: "84de40f7bd6d46e8afd909924d5b7083"
format_stamp: "Formatted at 2026-09-28 02:47:39 on dist-test-slave-b5p5"
server_key: "61bdcb74adb7292299949f40f1a452b6"
server_key_iv: "2408742ea0f9859be55b60a14dda0553"
server_key_version: "encryptionkey@0"
I20260928 02:47:39.870206 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:39.902091 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:39.902379 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:39.902568 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:39.902786 15507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:39.903142 15507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:39.903211 15507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.903249 15507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:39.903281 15507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:39.908973 15507 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.130:36085
I20260928 02:47:39.909019 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.130:36085 every 8 connection(s)
I20260928 02:47:39.909332 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/data/info.pb
I20260928 02:47:39.911424 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15507
I20260928 02:47:39.911536 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal/instance
I20260928 02:47:39.911707 28086 external_mini_cluster.cc:1428] Setting key 4b97e15e879d0308b3beb56adb8e789c
I20260928 02:47:39.912599 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.131:0
--local_ip_for_outbound_sockets=127.27.109.131
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--builtin_ntp_servers=127.27.109.148:37907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:47:39.914544 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.910018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.130:44607 (local address 127.27.109.190:43909)
0928 02:47:39.910082 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:39.910084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:39.910295 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:47:39.910845 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:39.910846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:39.910856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:39.910913 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:39.911548 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.912133 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.913185 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.913391 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.913476 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.914246 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:39.914247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:39.914248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:39.914252 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:39.914267 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:39.914271 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:39.914323 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:39.914421 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:47:39.914448 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:47:39.914780 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:39.910001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:43909 (local address 127.27.109.130:44607)
0928 02:47:39.910165 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:39.910211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:39.910735 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:39.910964 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:39.910967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:39.910975 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:39.911447 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.911451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.912228 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.912229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.913085 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:39.913087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:39.913485 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:39.913486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:39.913614 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:39.914089 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:39.914089 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:39.914115 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:39.914136 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:39.914334 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:39.914339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:39.914342 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:39.914423 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:47:39.914559 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260928 02:47:39.915259 15635 heartbeater.cc:347] Connected to a master server at 127.27.109.190:43909
I20260928 02:47:39.915359 15635 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:39.915541 15635 heartbeater.cc:511] Master 127.27.109.190:43909 requested a full tablet report, sending...
I20260928 02:47:39.915943 15316 ts_manager.cc:194] Registered new tserver with Master: 84de40f7bd6d46e8afd909924d5b7083 (127.27.109.130:36085)
I20260928 02:47:39.916461 15316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.130:44607
W20260928 02:47:39.992606 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:39.992890 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:39.992926 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:39.994329 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:39.994377 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:39.994412 15638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.131
I20260928 02:47:39.996385 15638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:37907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.131
--webserver_port=0
--tserver_master_addrs=127.27.109.190:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.131
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:39.996609 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:39.996841 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:40.001494 15646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:40.001514 15644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:40.001514 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:40.001875 15638 server_base.cc:1034] running on GCE node
I20260928 02:47:40.002072 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:40.002326 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:40.003489 15638 hybrid_clock.cc:648] HybridClock initialized: now 1790563660003482 us; error 37 us; skew 500 ppm
I20260928 02:47:40.004670 15638 webserver.cc:460] Webserver started at http://127.27.109.131:34307/ using document root <none> and password file <none>
I20260928 02:47:40.004913 15638 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:40.004992 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:40.005142 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:40.006129 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data/instance:
uuid: "a45ae5951dac4ebaa7e13e489bd24df5"
format_stamp: "Formatted at 2026-09-28 02:47:40 on dist-test-slave-b5p5"
server_key: "894f18d4c057fa2bd7b5d4d07e161124"
server_key_iv: "f85aac8a9ee03bd1fc6a6e15ac19a7cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:40.006533 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance:
uuid: "a45ae5951dac4ebaa7e13e489bd24df5"
format_stamp: "Formatted at 2026-09-28 02:47:40 on dist-test-slave-b5p5"
server_key: "894f18d4c057fa2bd7b5d4d07e161124"
server_key_iv: "f85aac8a9ee03bd1fc6a6e15ac19a7cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:40.007748 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:47:40.008415 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:40.008579 15638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:40.008724 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal
uuid: "a45ae5951dac4ebaa7e13e489bd24df5"
format_stamp: "Formatted at 2026-09-28 02:47:40 on dist-test-slave-b5p5"
server_key: "894f18d4c057fa2bd7b5d4d07e161124"
server_key_iv: "f85aac8a9ee03bd1fc6a6e15ac19a7cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:40.008829 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:40.040512 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:40.040797 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:40.040943 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:40.041172 15638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:47:40.041491 15638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:47:40.041543 15638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:40.041585 15638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:47:40.041613 15638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:40.047122 15638 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.131:33771
I20260928 02:47:40.047156 15765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.131:33771 every 8 connection(s)
I20260928 02:47:40.047480 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/data/info.pb
I20260928 02:47:40.047837 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15638
I20260928 02:47:40.047950 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal/instance
I20260928 02:47:40.048164 28086 external_mini_cluster.cc:1428] Setting key a36532feea7dd001fd9ffefa543c3b0e
I20260928 02:47:40.052419 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.048251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.131:39143 (local address 127.27.109.190:43909)
0928 02:47:40.048330 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.048332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.048547 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:47:40.049204 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.049204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.049214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.049258 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:40.049857 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.050319 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.051279 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.051464 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.051516 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.052194 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:40.052195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:40.052196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:40.052200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:40.052211 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:40.052215 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:40.052258 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.052344 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.052363 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:47:40.052629 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.048216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:43909 (local address 127.27.109.131:39143)
0928 02:47:40.048434 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.048474 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.049106 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.049315 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.049318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.049326 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:47:40.049782 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.049786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.050395 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.050396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.051207 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.051210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.051536 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.051536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.051636 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.052075 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:47:40.052076 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:47:40.052101 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:47:40.052122 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:47:40.052275 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:47:40.052279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:47:40.052281 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.052331 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.052401 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260928 02:47:40.053082 15766 heartbeater.cc:347] Connected to a master server at 127.27.109.190:43909
I20260928 02:47:40.053175 15766 heartbeater.cc:464] Registering TS with master...
I20260928 02:47:40.053337 15766 heartbeater.cc:511] Master 127.27.109.190:43909 requested a full tablet report, sending...
I20260928 02:47:40.053666 15316 ts_manager.cc:194] Registered new tserver with Master: a45ae5951dac4ebaa7e13e489bd24df5 (127.27.109.131:33771)
I20260928 02:47:40.054023 15316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.131:39143
I20260928 02:47:40.059857 28086 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:47:40.063627 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.061199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.27.109.190:43909)
0928 02:47:40.061248 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.061249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.061344 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:47:40.061376 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.061377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.061384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.061432 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:40.061564 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.061999 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.062729 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.062903 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.062953 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.063466 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:40.063467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:40.063468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:40.063471 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:40.063482 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:40.063486 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:40.063527 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.063565 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.063581 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:47:40.065203 15316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:47:40.066823 15316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:47:40.072945 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.068772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:33771 (local address 127.0.0.1:45268)
0928 02:47:40.069277 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.069286 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.069368 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.069541 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.069542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.069544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:40.069579 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.069580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070877 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.070878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.071842 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.071844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.072657 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.072658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.072754 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.072764 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.072860 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.072897 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260928 02:47:40.072986 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.068947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.27.109.131:33771)
0928 02:47:40.069099 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.069103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.069359 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:47:40.069443 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.069446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.069461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.069531 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:40.069656 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070818 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.071937 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.072579 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.072654 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.072729 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.072877 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.072912 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260928 02:47:40.073493 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.068772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:46655 (local address 127.0.0.1:33520)
0928 02:47:40.068924 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.068958 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.069020 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.069455 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.069457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.069459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:40.069515 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.069517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070815 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.070816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.072157 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.072160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.073300 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.073301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.073372 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.073376 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.073416 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.073442 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":100,"thread_start_us":100,"threads_started":1}
I20260928 02:47:40.073694 15700 tablet_service.cc:1467] Processing CreateTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 (DEFAULT_TABLE table=test-table [id=25fd4f55d89c437286558160f9d72dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:40.074045 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de84baf64fc4ad08e08b6d91c0efd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:40.074185 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.068780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:36085 (local address 127.0.0.1:52424)
0928 02:47:40.069619 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.069628 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.069671 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.069926 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.069927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.069929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:40.069968 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.069969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070864 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.070864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.072168 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.072170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.073940 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.073941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.073996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.073999 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.074078 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.074117 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":76,"thread_start_us":89,"threads_started":1}
I20260928 02:47:40.074285 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.069127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.27.109.129:46655)
0928 02:47:40.069292 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.069296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.069305 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:47:40.069345 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.069347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.069358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.069448 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:40.069597 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070733 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.072252 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.072999 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.074063 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.074146 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.074173 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.074212 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260928 02:47:40.074556 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.069006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.27.109.130:36085)
0928 02:47:40.069786 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.069789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.069796 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:47:40.069825 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.069826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.069843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.069928 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:40.070024 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.070750 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.073118 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.073852 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.073931 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.074011 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.074162 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.074208 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":81,"threads_started":1}
I20260928 02:47:40.075122 15569 tablet_service.cc:1467] Processing CreateTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 (DEFAULT_TABLE table=test-table [id=25fd4f55d89c437286558160f9d72dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:40.075430 15569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de84baf64fc4ad08e08b6d91c0efd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:40.076968 15780 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Bootstrap starting.
I20260928 02:47:40.077368 15438 tablet_service.cc:1467] Processing CreateTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 (DEFAULT_TABLE table=test-table [id=25fd4f55d89c437286558160f9d72dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:47:40.077667 15780 tablet_bootstrap.cc:654] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:40.077677 15438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de84baf64fc4ad08e08b6d91c0efd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:40.078011 15780 log.cc:824] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:40.078079 15781 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Bootstrap starting.
I20260928 02:47:40.078712 15781 tablet_bootstrap.cc:654] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:40.078862 15780 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: No bootstrap required, opened a new log
I20260928 02:47:40.078933 15780 ts_tablet_manager.cc:1397] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:40.079066 15781 log.cc:824] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:40.079725 15781 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: No bootstrap required, opened a new log
I20260928 02:47:40.079806 15781 ts_tablet_manager.cc:1397] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:40.080941 15784 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: Bootstrap starting.
I20260928 02:47:40.081411 15780 raft_consensus.cc:348] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.081557 15780 raft_consensus.cc:374] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:40.081600 15780 raft_consensus.cc:729] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45ae5951dac4ebaa7e13e489bd24df5, State: Initialized, Role: FOLLOWER
I20260928 02:47:40.081636 15784 tablet_bootstrap.cc:654] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:40.081705 15780 consensus_queue.cc:260] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.081947 15784 log.cc:824] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:40.081951 15780 ts_tablet_manager.cc:1428] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:47:40.081962 15766 heartbeater.cc:503] Master 127.27.109.190:43909 was elected leader, sending a full tablet report...
I20260928 02:47:40.082273 15781 raft_consensus.cc:348] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.082425 15781 raft_consensus.cc:374] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:40.082494 15781 raft_consensus.cc:729] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84de40f7bd6d46e8afd909924d5b7083, State: Initialized, Role: FOLLOWER
I20260928 02:47:40.082610 15781 consensus_queue.cc:260] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.082732 15784 tablet_bootstrap.cc:492] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: No bootstrap required, opened a new log
I20260928 02:47:40.082800 15784 ts_tablet_manager.cc:1397] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:47:40.082870 15781 ts_tablet_manager.cc:1428] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:47:40.083038 15635 heartbeater.cc:503] Master 127.27.109.190:43909 was elected leader, sending a full tablet report...
I20260928 02:47:40.084662 15784 raft_consensus.cc:348] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.084780 15784 raft_consensus.cc:374] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:40.084801 15784 raft_consensus.cc:729] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4abad56d4cac483783dc4f6fad695e73, State: Initialized, Role: FOLLOWER
I20260928 02:47:40.084929 15784 consensus_queue.cc:260] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.085192 15784 ts_tablet_manager.cc:1428] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:47:40.085199 15504 heartbeater.cc:503] Master 127.27.109.190:43909 was elected leader, sending a full tablet report...
I20260928 02:47:40.119202 15788 raft_consensus.cc:482] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:47:40.119285 15788 raft_consensus.cc:504] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.119634 15788 leader_election.cc:288] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a45ae5951dac4ebaa7e13e489bd24df5 (127.27.109.131:33771), 84de40f7bd6d46e8afd909924d5b7083 (127.27.109.130:36085)
I20260928 02:47:40.123296 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.119868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:34375 (local address 127.27.109.131:33771)
0928 02:47:40.119931 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.119933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.120048 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:47:40.120093 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.120094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.120101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.120163 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:40.120283 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.120954 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.122400 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.122960 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.123035 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.123042 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.123200 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.123234 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:47:40.123296 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.119855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.129:47721 (local address 127.27.109.130:36085)
0928 02:47:40.119897 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.119898 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.119909 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:47:40.119939 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.119940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.119950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.120014 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:47:40.120171 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.120922 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.122400 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.122960 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.123035 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.123042 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.123200 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.123235 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:47:40.123310 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.119786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.130:36085 (local address 127.27.109.129:47721)
0928 02:47:40.119838 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.119850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.119889 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.120036 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.120037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.120038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:40.120102 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.120103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.121020 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.121021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.122304 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.122307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.123043 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.123044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.123120 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.123123 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.123182 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.123228 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":67}
I20260928 02:47:40.123317 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.119800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.131:33771 (local address 127.27.109.129:34375)
0928 02:47:40.119959 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.119972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:47:40.120035 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:47:40.120168 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:47:40.120168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:47:40.120170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:47:40.120221 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.120222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.121020 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.121021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.122304 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:47:40.122307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.123045 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.123046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:47:40.123124 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.123126 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:47:40.123182 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:47:40.123227 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":57,"thread_start_us":89,"threads_started":1}
I20260928 02:47:40.123672 15589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de84baf64fc4ad08e08b6d91c0efd14" candidate_uuid: "4abad56d4cac483783dc4f6fad695e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84de40f7bd6d46e8afd909924d5b7083" is_pre_election: true
I20260928 02:47:40.123812 15589 raft_consensus.cc:2454] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abad56d4cac483783dc4f6fad695e73 in term 0.
I20260928 02:47:40.123847 15720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de84baf64fc4ad08e08b6d91c0efd14" candidate_uuid: "4abad56d4cac483783dc4f6fad695e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" is_pre_election: true
I20260928 02:47:40.123981 15720 raft_consensus.cc:2454] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4abad56d4cac483783dc4f6fad695e73 in term 0.
I20260928 02:47:40.124035 15394 leader_election.cc:302] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abad56d4cac483783dc4f6fad695e73, 84de40f7bd6d46e8afd909924d5b7083; no voters:
I20260928 02:47:40.124152 15788 raft_consensus.cc:2781] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:47:40.124213 15788 raft_consensus.cc:482] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:47:40.124230 15788 raft_consensus.cc:3037] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:40.124871 15788 raft_consensus.cc:504] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.125007 15788 leader_election.cc:288] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [CANDIDATE]: Term 1 election: Requested vote from peers a45ae5951dac4ebaa7e13e489bd24df5 (127.27.109.131:33771), 84de40f7bd6d46e8afd909924d5b7083 (127.27.109.130:36085)
I20260928 02:47:40.125134 15720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de84baf64fc4ad08e08b6d91c0efd14" candidate_uuid: "4abad56d4cac483783dc4f6fad695e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45ae5951dac4ebaa7e13e489bd24df5"
I20260928 02:47:40.125185 15589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de84baf64fc4ad08e08b6d91c0efd14" candidate_uuid: "4abad56d4cac483783dc4f6fad695e73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84de40f7bd6d46e8afd909924d5b7083"
I20260928 02:47:40.125203 15720 raft_consensus.cc:3037] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:40.125263 15589 raft_consensus.cc:3037] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:40.126055 15589 raft_consensus.cc:2454] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abad56d4cac483783dc4f6fad695e73 in term 1.
I20260928 02:47:40.126084 15720 raft_consensus.cc:2454] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4abad56d4cac483783dc4f6fad695e73 in term 1.
I20260928 02:47:40.126232 15394 leader_election.cc:302] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4abad56d4cac483783dc4f6fad695e73, 84de40f7bd6d46e8afd909924d5b7083; no voters:
I20260928 02:47:40.126333 15788 raft_consensus.cc:2781] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:40.126507 15788 raft_consensus.cc:686] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 1 LEADER]: Becoming Leader. State: Replica: 4abad56d4cac483783dc4f6fad695e73, State: Running, Role: LEADER
I20260928 02:47:40.126602 15788 consensus_queue.cc:237] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } }
I20260928 02:47:40.127427 15316 catalog_manager.cc:5495] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4abad56d4cac483783dc4f6fad695e73 (127.27.109.129). New cstate: current_term: 1 leader_uuid: "4abad56d4cac483783dc4f6fad695e73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4abad56d4cac483783dc4f6fad695e73" member_type: VOTER last_known_addr { host: "127.27.109.129" port: 46655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 } health_report { overall_health: UNKNOWN } } }
I20260928 02:47:40.145424 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.143324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.27.109.129:46655)
0928 02:47:40.143372 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:40.143373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:40.143380 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:47:40.143448 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:40.143448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:40.143463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:40.143510 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:47:40.143610 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.144030 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.144778 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:40.144950 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:40.144997 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:40.145122 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:47:40.145205 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:47:40.145259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:40.145296 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 02:47:40.145319 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:47:40.147001 15589 raft_consensus.cc:1261] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 1 FOLLOWER]: Refusing update from remote peer 4abad56d4cac483783dc4f6fad695e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:40.147007 15720 raft_consensus.cc:1261] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 1 FOLLOWER]: Refusing update from remote peer 4abad56d4cac483783dc4f6fad695e73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:47:40.147338 15790 consensus_queue.cc:1035] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84de40f7bd6d46e8afd909924d5b7083" member_type: VOTER last_known_addr { host: "127.27.109.130" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:40.147446 15788 consensus_queue.cc:1035] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a45ae5951dac4ebaa7e13e489bd24df5" member_type: VOTER last_known_addr { host: "127.27.109.131" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:47:40.149616 15794 mvcc.cc:205] Tried to move back new op lower bound from 7334148751959470080 to 7334148751879544832. Current Snapshot: MvccSnapshot[applied={T|T < 7334148751959470080}]
I20260928 02:47:40.151240 15795 mvcc.cc:205] Tried to move back new op lower bound from 7334148751959470080 to 7334148751879544832. Current Snapshot: MvccSnapshot[applied={T|T < 7334148751959470080 or (T in {7334148751959470080})}]
I20260928 02:47:40.151484 15316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:40.151587 15316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:47:40.152299 15316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c999cbe95af3418b88869981fca0d084: Sending DeleteTablet for 3 replicas of tablet 6de84baf64fc4ad08e08b6d91c0efd14
I20260928 02:47:40.152580 15438 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:40 UTC) from {username='slave'} at 127.0.0.1:33520
I20260928 02:47:40.152648 15569 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:40 UTC) from {username='slave'} at 127.0.0.1:52424
I20260928 02:47:40.152787 15700 tablet_service.cc:1514] Processing DeleteTablet for tablet 6de84baf64fc4ad08e08b6d91c0efd14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:47:40 UTC) from {username='slave'} at 127.0.0.1:45268
I20260928 02:47:40.152827 15803 tablet_replica.cc:330] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73: stopping tablet replica
I20260928 02:47:40.152911 15803 raft_consensus.cc:2229] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:47:40.152932 15804 tablet_replica.cc:330] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: stopping tablet replica
I20260928 02:47:40.153002 15803 raft_consensus.cc:2258] T 6de84baf64fc4ad08e08b6d91c0efd14 P 4abad56d4cac483783dc4f6fad695e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:40.153064 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15376
I20260928 02:47:40.154189 15804 raft_consensus.cc:2229] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:40.154305 15804 raft_consensus.cc:2258] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:40.154757 15804 ts_tablet_manager.cc:1905] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:40.155222 15805 tablet_replica.cc:330] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: stopping tablet replica
I20260928 02:47:40.155330 15805 raft_consensus.cc:2229] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:47:40.155447 15805 raft_consensus.cc:2258] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:47:40.155814 15805 ts_tablet_manager.cc:1905] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:47:40.156662 15804 ts_tablet_manager.cc:1918] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:40.156736 15804 log.cc:1196] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-1/wal/wals/6de84baf64fc4ad08e08b6d91c0efd14
I20260928 02:47:40.156980 15804 ts_tablet_manager.cc:1939] T 6de84baf64fc4ad08e08b6d91c0efd14 P 84de40f7bd6d46e8afd909924d5b7083: Deleting consensus metadata
I20260928 02:47:40.156989 15805 ts_tablet_manager.cc:1918] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:47:40.157065 15805 log.cc:1196] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Deleting WAL directory at /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563399140758-28086-0/minicluster-data/ts-2/wal/wals/6de84baf64fc4ad08e08b6d91c0efd14
I20260928 02:47:40.157311 15805 ts_tablet_manager.cc:1939] T 6de84baf64fc4ad08e08b6d91c0efd14 P a45ae5951dac4ebaa7e13e489bd24df5: Deleting consensus metadata
I20260928 02:47:40.157390 15303 catalog_manager.cc:4841] TS 84de40f7bd6d46e8afd909924d5b7083 (127.27.109.130:36085): tablet 6de84baf64fc4ad08e08b6d91c0efd14 (table test-table [id=25fd4f55d89c437286558160f9d72dc3]) successfully deleted
I20260928 02:47:40.157706 15300 catalog_manager.cc:4841] TS a45ae5951dac4ebaa7e13e489bd24df5 (127.27.109.131:33771): tablet 6de84baf64fc4ad08e08b6d91c0efd14 (table test-table [id=25fd4f55d89c437286558160f9d72dc3]) successfully deleted
W20260928 02:47:40.159281 15302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.129:46655 (error 108)
I20260928 02:47:40.159518 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15507
I20260928 02:47:40.159734 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:40.159612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.129:46655 (local address 127.0.0.1:33530)
0928 02:47:40.159652 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:47:40.159701 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.129:46655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 02:47:40.161355 15302 catalog_manager.cc:4584] TS 4abad56d4cac483783dc4f6fad695e73 (127.27.109.129:46655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6de84baf64fc4ad08e08b6d91c0efd14: Network error: Client connection negotiation failed: client connection to 127.27.109.129:46655: connect: Connection refused (error 111)
I20260928 02:47:40.164364 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15638
I20260928 02:47:40.169173 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15285
2026-09-28T02:47:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (687 ms)
[----------] 6 tests from EncryptionPolicyTest (8076 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:40 dist-test-slave-b5p5 krb5kdc[15809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:40 dist-test-slave-b5p5 krb5kdc[15809](info): set up 2 sockets
Sep 28 02:47:40 dist-test-slave-b5p5 krb5kdc[15809](info): commencing operation
krb5kdc: starting...
W20260928 02:47:41.198331 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:41.309803 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:41.388616 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:41.388857 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:41.388892 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:41.390259 15822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:41.390347 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:41.390374 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:41.390404 15822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:41.390434 15822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:41.391974 15822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:41.392239 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:41.392458 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:41.397082 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:41.397089 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:41.397090 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:41.397300 15822 server_base.cc:1034] running on GCE node
I20260928 02:47:41.397655 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:41.397944 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:41.399102 15822 hybrid_clock.cc:648] HybridClock initialized: now 1790563661399081 us; error 33 us; skew 500 ppm
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:41.401492 15822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:41.401895 15822 webserver.cc:460] Webserver started at http://127.0.0.1:42829/ using document root <none> and password file <none>
I20260928 02:47:41.402107 15822 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:41.402179 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:41.402326 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:41.403357 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "db115920b10041f09ede896264987064"
format_stamp: "Formatted at 2026-09-28 02:47:41 on dist-test-slave-b5p5"
server_key: "9d1e7fc3ea2f7260b97c93f39433e041"
server_key_iv: "25221fc72ceeb55b0e41c140ed455c9c"
server_key_version: "encryptionkey@0"
I20260928 02:47:41.403721 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "db115920b10041f09ede896264987064"
format_stamp: "Formatted at 2026-09-28 02:47:41 on dist-test-slave-b5p5"
server_key: "9d1e7fc3ea2f7260b97c93f39433e041"
server_key_iv: "25221fc72ceeb55b0e41c140ed455c9c"
server_key_version: "encryptionkey@0"
I20260928 02:47:41.404978 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:41.405666 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:41.405812 15822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:41.405907 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "db115920b10041f09ede896264987064"
format_stamp: "Formatted at 2026-09-28 02:47:41 on dist-test-slave-b5p5"
server_key: "9d1e7fc3ea2f7260b97c93f39433e041"
server_key_iv: "25221fc72ceeb55b0e41c140ed455c9c"
server_key_version: "encryptionkey@0"
I20260928 02:47:41.405983 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:41.425069 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:41.427392 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:41.427546 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:41.431434 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40653
I20260928 02:47:41.431464 15889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40653 every 8 connection(s)
I20260928 02:47:41.431806 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:41.432627 15890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:41.433880 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15822
I20260928 02:47:41.434010 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:41.434180 28086 external_mini_cluster.cc:1428] Setting key b73455e9c005584a9356b9d9be19ca6b
I20260928 02:47:41.435310 15890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064: Bootstrap starting.
I20260928 02:47:41.435981 15890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:41.436268 15890 log.cc:824] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:41.437012 15890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064: No bootstrap required, opened a new log
I20260928 02:47:41.439200 15890 raft_consensus.cc:348] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260928 02:47:41.439376 15890 raft_consensus.cc:374] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:41.439415 15890 raft_consensus.cc:729] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db115920b10041f09ede896264987064, State: Initialized, Role: FOLLOWER
I20260928 02:47:41.439556 15890 consensus_queue.cc:260] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260928 02:47:41.439658 15890 raft_consensus.cc:388] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:41.439697 15890 raft_consensus.cc:482] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:41.439746 15890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:41.440236 15890 raft_consensus.cc:504] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260928 02:47:41.440357 15890 leader_election.cc:302] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db115920b10041f09ede896264987064; no voters:
I20260928 02:47:41.440527 15890 leader_election.cc:288] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:41.440553 15895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:41.440665 15895 raft_consensus.cc:686] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [term 1 LEADER]: Becoming Leader. State: Replica: db115920b10041f09ede896264987064, State: Running, Role: LEADER
I20260928 02:47:41.440742 15895 consensus_queue.cc:237] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } }
I20260928 02:47:41.440770 15890 sys_catalog.cc:564] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:41.441051 15895 sys_catalog.cc:455] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db115920b10041f09ede896264987064. Latest consensus state: current_term: 1 leader_uuid: "db115920b10041f09ede896264987064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } } }
I20260928 02:47:41.441131 15895 sys_catalog.cc:458] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:41.441066 15897 sys_catalog.cc:455] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db115920b10041f09ede896264987064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db115920b10041f09ede896264987064" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40653 } } }
I20260928 02:47:41.441282 15897 sys_catalog.cc:458] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:41.441466 15900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:47:41.441983 15900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:41.444142 15900 catalog_manager.cc:1347] Generated new cluster ID: 1d727a51c10a444e8586bc8a3d0f6707
I20260928 02:47:41.444192 15900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:41.445847 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:41.434726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41643 (local address 127.0.0.1:40653)
0928 02:47:41.434963 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:41.434972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:41.435004 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:47:41.435513 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:41.435522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:41.435576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:41.435744 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:41.436489 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:41.437810 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:41.438641 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:41.439300 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:41.439412 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:41.442344 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:41.442348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:41.442349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:41.442361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:41.444220 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:41.444642 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:41.444645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:41.444645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:41.444732 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:41.444943 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:41.444943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:41.444943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:41.445200 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:41.445308 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:41.445553 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:47:41.445644 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":129,"threads_started":1}
I20260928 02:47:41.447376 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:41.454542 15900 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:41.455331 15900 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:41.457441 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:41.448751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48397 (local address 127.0.0.1:40653)
0928 02:47:41.448780 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:41.448781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:41.450605 (+ 1824us) server_negotiation.cc:407] Connection header received
0928 02:47:41.450703 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:41.450704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:41.450714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:41.450777 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:41.451344 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:41.452098 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:41.453064 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:41.453404 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:41.453473 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:41.454734 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:41.454736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:41.454736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:41.454743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:41.456284 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:41.456676 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:41.456677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:41.456677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:41.456735 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:41.456954 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:41.456955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:41.456956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:41.457076 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:41.457187 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:41.457298 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:47:41.457357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:47:41.465904 15900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db115920b10041f09ede896264987064: Generated new TSK 0
I20260928 02:47:41.466081 15900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:41.470189 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15822
W20260928 02:47:41.474327 15807 connection.cc:535] client connection to 127.0.0.1:40653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40653): Connection reset by peer (error 104)
2026-09-28T02:47:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15921](info): set up 2 sockets
Sep 28 02:47:41 dist-test-slave-b5p5 krb5kdc[15921](info): commencing operation
krb5kdc: starting...
W20260928 02:47:42.498327 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:42 dist-test-slave-b5p5 krb5kdc[15921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:42.607199 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:42.683833 15934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:42.684048 15934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:42.684090 15934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:42.685194 15934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:42.685259 15934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:42.685284 15934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:42.685307 15934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:42.685345 15934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:42.686791 15934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:42.687057 15934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:42.687268 15934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:42.691902 15939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:42.691916 15942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:42.691916 15940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:42.692039 15934 server_base.cc:1034] running on GCE node
I20260928 02:47:42.692323 15934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:42.692549 15934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:42.693687 15934 hybrid_clock.cc:648] HybridClock initialized: now 1790563662693658 us; error 45 us; skew 500 ppm
Sep 28 02:47:42 dist-test-slave-b5p5 krb5kdc[15921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:42.696053 15934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:42.696455 15934 webserver.cc:460] Webserver started at http://127.0.0.1:44763/ using document root <none> and password file <none>
I20260928 02:47:42.696660 15934 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:42.696704 15934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:42.696863 15934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:42.697851 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "9dbfb4c409924b12a440412728e7b1ca"
format_stamp: "Formatted at 2026-09-28 02:47:42 on dist-test-slave-b5p5"
server_key: "27e0074855e2b13e45bbfd7289da35dc"
server_key_iv: "21c21ebc9605994ee5502f1111cd9aca"
server_key_version: "encryptionkey@0"
I20260928 02:47:42.698185 15934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "9dbfb4c409924b12a440412728e7b1ca"
format_stamp: "Formatted at 2026-09-28 02:47:42 on dist-test-slave-b5p5"
server_key: "27e0074855e2b13e45bbfd7289da35dc"
server_key_iv: "21c21ebc9605994ee5502f1111cd9aca"
server_key_version: "encryptionkey@0"
I20260928 02:47:42.699512 15934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:42.700202 15949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:42.700381 15934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:42.700467 15934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "9dbfb4c409924b12a440412728e7b1ca"
format_stamp: "Formatted at 2026-09-28 02:47:42 on dist-test-slave-b5p5"
server_key: "27e0074855e2b13e45bbfd7289da35dc"
server_key_iv: "21c21ebc9605994ee5502f1111cd9aca"
server_key_version: "encryptionkey@0"
I20260928 02:47:42.700537 15934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:42.708575 15934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:42.710736 15934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:42.710884 15934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:42.714383 15934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46335
I20260928 02:47:42.714450 16001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46335 every 8 connection(s)
I20260928 02:47:42.714745 15934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:42.715487 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:42.717455 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: Bootstrap starting.
I20260928 02:47:42.717978 16002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:42.718254 16002 log.cc:824] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:42.719002 16002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: No bootstrap required, opened a new log
I20260928 02:47:42.720505 16002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260928 02:47:42.720624 16002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:42.720646 16002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbfb4c409924b12a440412728e7b1ca, State: Initialized, Role: FOLLOWER
I20260928 02:47:42.720845 16002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260928 02:47:42.720927 16002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:42.720968 16002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:42.721014 16002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:42.721017 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 15934
I20260928 02:47:42.721123 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:42.721307 28086 external_mini_cluster.cc:1428] Setting key 0dca2d627fc89b146f91d758a3f01ff6
I20260928 02:47:42.721648 16002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260928 02:47:42.721756 16002 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbfb4c409924b12a440412728e7b1ca; no voters:
I20260928 02:47:42.721909 16002 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:42.721935 16007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:42.722093 16007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [term 1 LEADER]: Becoming Leader. State: Replica: 9dbfb4c409924b12a440412728e7b1ca, State: Running, Role: LEADER
I20260928 02:47:42.722136 16002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:42.722193 16007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260928 02:47:42.722530 16007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbfb4c409924b12a440412728e7b1ca. Latest consensus state: current_term: 1 leader_uuid: "9dbfb4c409924b12a440412728e7b1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260928 02:47:42.722671 16007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:42.722676 16009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbfb4c409924b12a440412728e7b1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbfb4c409924b12a440412728e7b1ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260928 02:47:42.722808 16009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca [sys.catalog]: This master's current role is: LEADER
W20260928 02:47:42.723932 16023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:47:42.723985 16023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:47:42.724026 16016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:42.724164 16016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:42.725592 16016 catalog_manager.cc:1347] Generated new cluster ID: 4b829351184f4580b22927bca1ceab5a
I20260928 02:47:42.725646 16016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:42.727780 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:42.721758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53787 (local address 127.0.0.1:46335)
0928 02:47:42.722914 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:42.722922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:42.722961 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:47:42.723042 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:42.723066 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:42.723117 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:42.723532 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:42.723695 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:42.724920 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:42.725983 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:42.726387 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:42.726519 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:42.727179 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:42.727182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:42.727183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:42.727348 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:42.727375 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:42.727385 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:42.727480 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:42.727558 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:47:42.727584 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":84,"threads_started":1}
I20260928 02:47:42.728750 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:42.735213 16016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:42.735859 16016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:42.737434 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:42.729998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41963 (local address 127.0.0.1:46335)
0928 02:47:42.730031 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:42.730033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:42.734628 (+ 4595us) server_negotiation.cc:407] Connection header received
0928 02:47:42.734700 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:42.734701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:42.734711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:42.734759 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:42.734921 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:42.735573 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:42.736296 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:42.736525 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:42.736596 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:42.737198 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:42.737199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:42.737200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:42.737204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:42.737216 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:42.737220 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:42.737273 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:42.737332 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:47:42.737344 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:47:42.741844 16016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbfb4c409924b12a440412728e7b1ca: Generated new TSK 0
I20260928 02:47:42.742004 16016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:42.743916 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 15934
W20260928 02:47:42.747524 15919 connection.cc:535] client connection to 127.0.0.1:46335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46335): Connection reset by peer (error 104)
2026-09-28T02:47:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:42 dist-test-slave-b5p5 krb5kdc[16033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:42 dist-test-slave-b5p5 krb5kdc[16033](info): set up 2 sockets
Sep 28 02:47:42 dist-test-slave-b5p5 krb5kdc[16033](info): commencing operation
krb5kdc: starting...
W20260928 02:47:43.762423 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:43 dist-test-slave-b5p5 krb5kdc[16033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:43.871461 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:43.949177 16046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:43.949391 16046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:43.949426 16046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:43.950565 16046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:43.950632 16046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:43.950655 16046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:43.950675 16046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:43.950702 16046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:43.952154 16046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:43.952414 16046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:43.952620 16046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:43.957182 16051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:43.957191 16054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:43.957451 16046 server_base.cc:1034] running on GCE node
W20260928 02:47:43.957221 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:43.957756 16046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:43.957974 16046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:43.959098 16046 hybrid_clock.cc:648] HybridClock initialized: now 1790563663959068 us; error 46 us; skew 500 ppm
Sep 28 02:47:43 dist-test-slave-b5p5 krb5kdc[16033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:43.965546 16046 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:43.965987 16046 webserver.cc:460] Webserver started at http://127.0.0.1:41505/ using document root <none> and password file <none>
I20260928 02:47:43.966197 16046 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:43.966241 16046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:43.966401 16046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:43.967408 16046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "c914df21527c4335ace9a1d5b35c6e4b"
format_stamp: "Formatted at 2026-09-28 02:47:43 on dist-test-slave-b5p5"
server_key: "95698cf41f3bdbc08690a814d96181cc"
server_key_iv: "f7501b477b82cc5a6d95e49580e84a20"
server_key_version: "encryptionkey@0"
I20260928 02:47:43.967758 16046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "c914df21527c4335ace9a1d5b35c6e4b"
format_stamp: "Formatted at 2026-09-28 02:47:43 on dist-test-slave-b5p5"
server_key: "95698cf41f3bdbc08690a814d96181cc"
server_key_iv: "f7501b477b82cc5a6d95e49580e84a20"
server_key_version: "encryptionkey@0"
I20260928 02:47:43.969111 16046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:43.969834 16061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:43.970016 16046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:43.970115 16046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "c914df21527c4335ace9a1d5b35c6e4b"
format_stamp: "Formatted at 2026-09-28 02:47:43 on dist-test-slave-b5p5"
server_key: "95698cf41f3bdbc08690a814d96181cc"
server_key_iv: "f7501b477b82cc5a6d95e49580e84a20"
server_key_version: "encryptionkey@0"
I20260928 02:47:43.970197 16046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:43.973390 16046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:43.975173 16046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:43.975327 16046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:43.979151 16046 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45067
I20260928 02:47:43.979190 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45067 every 8 connection(s)
I20260928 02:47:43.979487 16046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:43.980361 16114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:43.982414 16114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b: Bootstrap starting.
I20260928 02:47:43.983002 16114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:43.983278 16114 log.cc:824] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:43.983924 16114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b: No bootstrap required, opened a new log
I20260928 02:47:43.985261 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16046
I20260928 02:47:43.985355 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:43.985558 28086 external_mini_cluster.cc:1428] Setting key bf43a6de3511f1eaacba823ef34babe6
I20260928 02:47:43.985544 16114 raft_consensus.cc:348] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } }
I20260928 02:47:43.985718 16114 raft_consensus.cc:374] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:43.985769 16114 raft_consensus.cc:729] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c914df21527c4335ace9a1d5b35c6e4b, State: Initialized, Role: FOLLOWER
I20260928 02:47:43.985910 16114 consensus_queue.cc:260] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } }
I20260928 02:47:43.985985 16114 raft_consensus.cc:388] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:43.986018 16114 raft_consensus.cc:482] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:43.986080 16114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:43.986729 16114 raft_consensus.cc:504] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } }
I20260928 02:47:43.986841 16114 leader_election.cc:302] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c914df21527c4335ace9a1d5b35c6e4b; no voters:
I20260928 02:47:43.987298 16114 leader_election.cc:288] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:43.987344 16119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:43.987483 16119 raft_consensus.cc:686] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [term 1 LEADER]: Becoming Leader. State: Replica: c914df21527c4335ace9a1d5b35c6e4b, State: Running, Role: LEADER
I20260928 02:47:43.987571 16114 sys_catalog.cc:564] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:43.987629 16119 consensus_queue.cc:237] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } }
I20260928 02:47:43.987975 16120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c914df21527c4335ace9a1d5b35c6e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } } }
I20260928 02:47:43.987965 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:43.986059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54411 (local address 127.0.0.1:45067)
0928 02:47:43.986221 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:43.986230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:43.986262 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:47:43.986387 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:43.986391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:43.986636 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:43.986749 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:43.987384 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:43.987388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:43.987388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:43.987392 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:43.987436 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:43.987453 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:43.987665 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:43.987697 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:47:43.987778 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260928 02:47:43.987964 16119 sys_catalog.cc:455] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c914df21527c4335ace9a1d5b35c6e4b. Latest consensus state: current_term: 1 leader_uuid: "c914df21527c4335ace9a1d5b35c6e4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c914df21527c4335ace9a1d5b35c6e4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45067 } } }
I20260928 02:47:43.988204 16119 sys_catalog.cc:458] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:43.988145 16120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:43.988934 16128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:43.989494 16128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:43.989495 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:43.990975 16128 catalog_manager.cc:1347] Generated new cluster ID: cf8d66a0076147728bb3fcfd7d73d582
I20260928 02:47:43.991027 16128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:43.995767 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:43.994758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42659 (local address 127.0.0.1:45067)
0928 02:47:43.994796 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:43.994798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:43.994866 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:47:43.994932 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:43.994933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:43.994937 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:43.994999 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:43.995554 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:43.995555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:43.995556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:43.995560 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:43.995575 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:43.995579 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:43.995639 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:43.995689 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:47:43.995701 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:47:44.001973 16128 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:44.002106 16128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:44.004066 16128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c914df21527c4335ace9a1d5b35c6e4b: Generated new TSK 0
I20260928 02:47:44.004212 16128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:44.006316 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16046
2026-09-28T02:47:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:44 dist-test-slave-b5p5 krb5kdc[16145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:44 dist-test-slave-b5p5 krb5kdc[16145](info): set up 2 sockets
Sep 28 02:47:44 dist-test-slave-b5p5 krb5kdc[16145](info): commencing operation
krb5kdc: starting...
W20260928 02:47:45.035905 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:45.148578 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:45.226771 16158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:45.226953 16158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:45.226974 16158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:45.228106 16158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:45.228147 16158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:45.228161 16158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:45.228174 16158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:45.228186 16158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:45.229619 16158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:45.229851 16158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:45.230100 16158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:45.234696 16163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:45.234694 16164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:45.234792 16158 server_base.cc:1034] running on GCE node
W20260928 02:47:45.234691 16166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:45.235096 16158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:45.235378 16158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:45.236625 16158 hybrid_clock.cc:648] HybridClock initialized: now 1790563665236619 us; error 34 us; skew 500 ppm
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:45.239110 16158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:45.239523 16158 webserver.cc:460] Webserver started at http://127.0.0.1:39575/ using document root <none> and password file <none>
I20260928 02:47:45.239732 16158 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:45.239795 16158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:45.239939 16158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:45.240906 16158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "3ca3595a3f7c435081c2a39065342615"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-b5p5"
server_key: "85fb4e58dfb4bb1351bf9aebf8ae301c"
server_key_iv: "2e9d828f5fa1b1255ad60357e87cdd0f"
server_key_version: "encryptionkey@0"
I20260928 02:47:45.241269 16158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "3ca3595a3f7c435081c2a39065342615"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-b5p5"
server_key: "85fb4e58dfb4bb1351bf9aebf8ae301c"
server_key_iv: "2e9d828f5fa1b1255ad60357e87cdd0f"
server_key_version: "encryptionkey@0"
I20260928 02:47:45.242654 16158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:47:45.243347 16173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:45.243517 16158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:45.243618 16158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "3ca3595a3f7c435081c2a39065342615"
format_stamp: "Formatted at 2026-09-28 02:47:45 on dist-test-slave-b5p5"
server_key: "85fb4e58dfb4bb1351bf9aebf8ae301c"
server_key_iv: "2e9d828f5fa1b1255ad60357e87cdd0f"
server_key_version: "encryptionkey@0"
I20260928 02:47:45.243688 16158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:45.253199 16158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:45.255009 16158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:45.255162 16158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:45.258837 16158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44211
I20260928 02:47:45.258939 16225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44211 every 8 connection(s)
I20260928 02:47:45.259238 16158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:45.260084 16226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:45.262223 16226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615: Bootstrap starting.
I20260928 02:47:45.262614 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16158
I20260928 02:47:45.262738 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:45.262918 28086 external_mini_cluster.cc:1428] Setting key afd16472f59e91397b95b0c1d2841a36
I20260928 02:47:45.262926 16226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:45.263207 16226 log.cc:824] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:45.263985 16226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615: No bootstrap required, opened a new log
I20260928 02:47:45.266247 16226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260928 02:47:45.266407 16226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:45.266449 16226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca3595a3f7c435081c2a39065342615, State: Initialized, Role: FOLLOWER
I20260928 02:47:45.266597 16226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260928 02:47:45.266674 16226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:45.266696 16226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:45.266773 16226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:45.267314 16226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260928 02:47:45.267414 16226 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca3595a3f7c435081c2a39065342615; no voters:
I20260928 02:47:45.267553 16226 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:45.267630 16231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:45.267782 16226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:45.267789 16231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [term 1 LEADER]: Becoming Leader. State: Replica: 3ca3595a3f7c435081c2a39065342615, State: Running, Role: LEADER
I20260928 02:47:45.267899 16231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260928 02:47:45.268168 16231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca3595a3f7c435081c2a39065342615. Latest consensus state: current_term: 1 leader_uuid: "3ca3595a3f7c435081c2a39065342615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260928 02:47:45.268275 16231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:45.268182 16232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca3595a3f7c435081c2a39065342615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca3595a3f7c435081c2a39065342615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260928 02:47:45.268440 16232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:45.268532 16236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:45.269037 16236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:47:45.270539 16236 catalog_manager.cc:1347] Generated new cluster ID: a9c9baf08d2b4324ad7cc374a4401982
I20260928 02:47:45.270588 16236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:45.274152 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:45.263349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58579 (local address 127.0.0.1:44211)
0928 02:47:45.263493 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:45.263501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:45.263531 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:47:45.264121 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:45.264130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:45.264188 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:45.264344 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:45.264937 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:45.266380 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:45.267303 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:45.267916 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:45.268358 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:45.271020 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:45.271024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:45.271024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:45.271035 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:45.272667 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:45.273044 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:45.273047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:45.273047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:45.273132 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:45.273336 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:45.273337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:45.273337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:45.273561 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:45.273733 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:45.273829 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:47:45.273960 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260928 02:47:45.275014 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:45.282119 16236 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:45.282790 16236 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:45.284963 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:45.278926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54263 (local address 127.0.0.1:44211)
0928 02:47:45.278966 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:45.278968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:45.279119 (+ 151us) server_negotiation.cc:407] Connection header received
0928 02:47:45.279191 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:45.279192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:45.279202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:45.279246 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:45.279767 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:45.280252 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:45.280957 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:45.281149 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:45.281223 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:45.282508 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:45.282510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:45.282510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:45.282517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:45.283852 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:45.284214 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:45.284215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:45.284216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:45.284271 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:45.284477 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:45.284478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:45.284478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:45.284580 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:45.284684 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:45.284782 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:47:45.284866 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:47:45.289744 16236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca3595a3f7c435081c2a39065342615: Generated new TSK 0
I20260928 02:47:45.289940 16236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:45.294970 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16158
2026-09-28T02:47:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16257](info): set up 2 sockets
Sep 28 02:47:45 dist-test-slave-b5p5 krb5kdc[16257](info): commencing operation
krb5kdc: starting...
W20260928 02:47:46.322336 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:46 dist-test-slave-b5p5 krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:46.430877 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:46.509231 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:46.509415 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:46.509438 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:46.510560 16270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:46.510599 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:46.510614 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:46.510664 16270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:46.510691 16270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:46.512089 16270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:46.512315 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:46.512524 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:46.517156 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:46.517208 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:46.517207 16270 server_base.cc:1034] running on GCE node
W20260928 02:47:46.517164 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:46.517681 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:46.517951 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:46.519105 16270 hybrid_clock.cc:648] HybridClock initialized: now 1790563666519086 us; error 35 us; skew 500 ppm
Sep 28 02:47:46 dist-test-slave-b5p5 krb5kdc[16257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:46.521317 16270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:46.521701 16270 webserver.cc:460] Webserver started at http://127.0.0.1:39865/ using document root <none> and password file <none>
I20260928 02:47:46.521914 16270 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:46.521981 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:46.522125 16270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:46.523168 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "a3e73af713604762b6a740146912862d"
format_stamp: "Formatted at 2026-09-28 02:47:46 on dist-test-slave-b5p5"
server_key: "40922ae2b35f94ca2510bad9e7e3845e"
server_key_iv: "513b8171c046f16e6aec457e02aeac7f"
server_key_version: "encryptionkey@0"
I20260928 02:47:46.523511 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "a3e73af713604762b6a740146912862d"
format_stamp: "Formatted at 2026-09-28 02:47:46 on dist-test-slave-b5p5"
server_key: "40922ae2b35f94ca2510bad9e7e3845e"
server_key_iv: "513b8171c046f16e6aec457e02aeac7f"
server_key_version: "encryptionkey@0"
I20260928 02:47:46.524763 16270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:46.525434 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:46.525586 16270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:46.525681 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "a3e73af713604762b6a740146912862d"
format_stamp: "Formatted at 2026-09-28 02:47:46 on dist-test-slave-b5p5"
server_key: "40922ae2b35f94ca2510bad9e7e3845e"
server_key_iv: "513b8171c046f16e6aec457e02aeac7f"
server_key_version: "encryptionkey@0"
I20260928 02:47:46.525774 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:46.542356 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:46.544072 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:46.544226 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:46.547859 16270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42209
I20260928 02:47:46.547914 16337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42209 every 8 connection(s)
I20260928 02:47:46.548249 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:46.549038 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:46.551185 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d: Bootstrap starting.
I20260928 02:47:46.551748 16338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:46.552031 16338 log.cc:824] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:46.552676 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d: No bootstrap required, opened a new log
I20260928 02:47:46.554229 16338 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } }
I20260928 02:47:46.554358 16338 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:46.554404 16338 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e73af713604762b6a740146912862d, State: Initialized, Role: FOLLOWER
I20260928 02:47:46.554596 16338 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } }
I20260928 02:47:46.554692 16338 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:46.554728 16338 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:46.554735 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16270
I20260928 02:47:46.554780 16338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:46.554852 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:46.555055 28086 external_mini_cluster.cc:1428] Setting key 6ab800c89975bee00f3a90f3cdc9ae74
I20260928 02:47:46.555454 16338 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } }
I20260928 02:47:46.555558 16338 leader_election.cc:302] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e73af713604762b6a740146912862d; no voters:
I20260928 02:47:46.555701 16338 leader_election.cc:288] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:46.555757 16343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:46.555895 16343 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [term 1 LEADER]: Becoming Leader. State: Replica: a3e73af713604762b6a740146912862d, State: Running, Role: LEADER
I20260928 02:47:46.555927 16338 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:46.555987 16343 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } }
I20260928 02:47:46.556523 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e73af713604762b6a740146912862d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } } }
I20260928 02:47:46.556622 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:46.556511 16343 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e73af713604762b6a740146912862d. Latest consensus state: current_term: 1 leader_uuid: "a3e73af713604762b6a740146912862d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e73af713604762b6a740146912862d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42209 } } }
I20260928 02:47:46.556795 16343 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:46.556867 16352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:46.557365 16352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:46.559048 16352 catalog_manager.cc:1347] Generated new cluster ID: 45b712f7aea5488fa682f4b5732b9548
I20260928 02:47:46.559099 16352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:46.560889 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:46.555487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39529 (local address 127.0.0.1:42209)
0928 02:47:46.555669 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:46.555677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:46.555710 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:47:46.555836 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:46.555843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:46.555893 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:46.556295 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:46.556390 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.557879 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.558928 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.559497 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.559593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:46.560243 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:46.560246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:46.560247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:46.560451 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:46.560485 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:46.560497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:46.560555 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:46.560644 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 02:47:46.560699 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260928 02:47:46.561702 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:46.566435 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:46.563109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40339 (local address 127.0.0.1:42209)
0928 02:47:46.563146 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:46.563147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:46.563300 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:47:46.563358 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:46.563359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:46.563377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:46.563428 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:46.563554 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.564224 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.565269 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:46.565550 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:46.565603 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:46.566213 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:46.566214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:46.566214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:46.566219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:46.566230 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:46.566234 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:46.566282 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:46.566348 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 02:47:46.566370 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:47:46.570721 16352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:46.571177 16352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:46.577574 16352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e73af713604762b6a740146912862d: Generated new TSK 0
I20260928 02:47:46.577737 16352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:46.582365 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16270
2026-09-28T02:47:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:46 dist-test-slave-b5p5 krb5kdc[16369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:46 dist-test-slave-b5p5 krb5kdc[16369](info): set up 2 sockets
Sep 28 02:47:46 dist-test-slave-b5p5 krb5kdc[16369](info): commencing operation
krb5kdc: starting...
W20260928 02:47:47.622574 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:47 dist-test-slave-b5p5 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:47.733081 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:47.810541 16382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:47.810746 16382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:47.810783 16382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:47.811877 16382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:47.811946 16382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:47.811970 16382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:47.812002 16382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:47.812028 16382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:47.813452 16382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:47.813702 16382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:47.813913 16382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:47.818627 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:47.818637 16388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.818848 16382 server_base.cc:1034] running on GCE node
W20260928 02:47:47.818637 16390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:47.819136 16382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:47.819394 16382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:47.820559 16382 hybrid_clock.cc:648] HybridClock initialized: now 1790563667820535 us; error 34 us; skew 500 ppm
Sep 28 02:47:47 dist-test-slave-b5p5 krb5kdc[16369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:47.822880 16382 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:47.823320 16382 webserver.cc:460] Webserver started at http://127.0.0.1:34457/ using document root <none> and password file <none>
I20260928 02:47:47.823544 16382 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:47.823611 16382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:47.823742 16382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:47.824687 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "fc7289a905db48fbac9297654730a66f"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-b5p5"
server_key: "059656b0eb11b1820e1126fe2b0f9a27"
server_key_iv: "f7e3e196ec894133efb5a457969da107"
server_key_version: "encryptionkey@0"
I20260928 02:47:47.825029 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "fc7289a905db48fbac9297654730a66f"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-b5p5"
server_key: "059656b0eb11b1820e1126fe2b0f9a27"
server_key_iv: "f7e3e196ec894133efb5a457969da107"
server_key_version: "encryptionkey@0"
I20260928 02:47:47.826252 16382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:47:47.827052 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.827224 16382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:47.827315 16382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "fc7289a905db48fbac9297654730a66f"
format_stamp: "Formatted at 2026-09-28 02:47:47 on dist-test-slave-b5p5"
server_key: "059656b0eb11b1820e1126fe2b0f9a27"
server_key_iv: "f7e3e196ec894133efb5a457969da107"
server_key_version: "encryptionkey@0"
I20260928 02:47:47.827400 16382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:47.830560 16382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:47.832207 16382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:47.832360 16382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:47.836167 16382 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39443
I20260928 02:47:47.836199 16449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39443 every 8 connection(s)
I20260928 02:47:47.836486 16382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:47.837071 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16382
I20260928 02:47:47.837224 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:47.837397 28086 external_mini_cluster.cc:1428] Setting key 2fbc7c9ac13b9ba8243b0cd40125b00d
I20260928 02:47:47.837582 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:47.839799 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.837888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48471 (local address 127.0.0.1:39443)
0928 02:47:47.838062 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.838070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.838103 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:47:47.838620 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.838627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.838650 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.838756 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.839355 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.839358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.839358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:47.839361 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.839394 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:47.839404 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.839464 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.839562 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.839590 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 02:47:47.840822 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:47.840906 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f: Bootstrap starting.
I20260928 02:47:47.841583 16450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:47.841908 16450 log.cc:824] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:47.842829 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f: No bootstrap required, opened a new log
I20260928 02:47:47.843380 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:47.842399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55649 (local address 127.0.0.1:39443)
0928 02:47:47.842435 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:47.842437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:47.842581 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:47:47.842622 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:47.842623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:47.842628 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:47.842686 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:47.843191 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:47.843192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:47.843193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:47.843197 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:47.843211 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:47.843215 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:47.843278 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:47.843313 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:47:47.843325 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:47:47.844822 16450 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260928 02:47:47.844970 16450 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:47.845018 16450 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7289a905db48fbac9297654730a66f, State: Initialized, Role: FOLLOWER
I20260928 02:47:47.845129 16450 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260928 02:47:47.845227 16450 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:47.845268 16450 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:47.845315 16450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:47.845803 16450 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260928 02:47:47.845914 16450 leader_election.cc:302] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7289a905db48fbac9297654730a66f; no voters:
I20260928 02:47:47.846066 16450 leader_election.cc:288] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:47.846102 16460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:47.846257 16460 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [term 1 LEADER]: Becoming Leader. State: Replica: fc7289a905db48fbac9297654730a66f, State: Running, Role: LEADER
I20260928 02:47:47.846297 16450 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:47.846400 16460 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } }
I20260928 02:47:47.846788 16461 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7289a905db48fbac9297654730a66f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } } }
I20260928 02:47:47.846791 16462 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7289a905db48fbac9297654730a66f. Latest consensus state: current_term: 1 leader_uuid: "fc7289a905db48fbac9297654730a66f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7289a905db48fbac9297654730a66f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39443 } } }
I20260928 02:47:47.846972 16461 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:47.847000 16462 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:47.847421 16469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:47.848066 16469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:47.849453 16469 catalog_manager.cc:1347] Generated new cluster ID: 6bd04ccc3004481fbce2ce710580ec31
I20260928 02:47:47.849506 16469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:47.863973 16469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:47.864144 16469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:47.869197 16469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7289a905db48fbac9297654730a66f: Generated new TSK 0
I20260928 02:47:47.869369 16469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:47.871148 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16382
2026-09-28T02:47:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:47 dist-test-slave-b5p5 krb5kdc[16481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:47 dist-test-slave-b5p5 krb5kdc[16481](info): set up 2 sockets
Sep 28 02:47:47 dist-test-slave-b5p5 krb5kdc[16481](info): commencing operation
krb5kdc: starting...
W20260928 02:47:48.898389 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:48 dist-test-slave-b5p5 krb5kdc[16481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:49.005105 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:49.090214 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:49.090458 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:49.090523 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:49.091758 16494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:49.091837 16494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:49.091866 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:49.091897 16494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:49.091921 16494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:49.093415 16494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:49.093678 16494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:49.093906 16494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:49.098812 16500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:49.098773 16502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:49.098869 16494 server_base.cc:1034] running on GCE node
W20260928 02:47:49.098773 16499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:49.099229 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:49.099529 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:49.100672 16494 hybrid_clock.cc:648] HybridClock initialized: now 1790563669100661 us; error 28 us; skew 500 ppm
Sep 28 02:47:49 dist-test-slave-b5p5 krb5kdc[16481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:49.107502 16494 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:49.107901 16494 webserver.cc:460] Webserver started at http://127.0.0.1:40209/ using document root <none> and password file <none>
I20260928 02:47:49.108091 16494 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:49.108131 16494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:49.108214 16494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:49.109222 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "2c721d0548b64bd894c491948cd64267"
format_stamp: "Formatted at 2026-09-28 02:47:49 on dist-test-slave-b5p5"
server_key: "e6b760809ca97e46a56d3e62ffab2dbc"
server_key_iv: "c0f3548c8997ff74fe76cd20bd7d38fe"
server_key_version: "encryptionkey@0"
I20260928 02:47:49.109588 16494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "2c721d0548b64bd894c491948cd64267"
format_stamp: "Formatted at 2026-09-28 02:47:49 on dist-test-slave-b5p5"
server_key: "e6b760809ca97e46a56d3e62ffab2dbc"
server_key_iv: "c0f3548c8997ff74fe76cd20bd7d38fe"
server_key_version: "encryptionkey@0"
I20260928 02:47:49.110939 16494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:49.111685 16509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:49.111893 16494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:49.112005 16494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "2c721d0548b64bd894c491948cd64267"
format_stamp: "Formatted at 2026-09-28 02:47:49 on dist-test-slave-b5p5"
server_key: "e6b760809ca97e46a56d3e62ffab2dbc"
server_key_iv: "c0f3548c8997ff74fe76cd20bd7d38fe"
server_key_version: "encryptionkey@0"
I20260928 02:47:49.112084 16494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:49.140446 16494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:49.142288 16494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:49.142484 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:49.146224 16494 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35971
I20260928 02:47:49.146260 16561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35971 every 8 connection(s)
I20260928 02:47:49.146605 16494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:49.147415 16562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:49.149528 16562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267: Bootstrap starting.
I20260928 02:47:49.150045 16562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:49.150054 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16494
I20260928 02:47:49.150334 16562 log.cc:824] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:49.150332 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:49.150614 28086 external_mini_cluster.cc:1428] Setting key cc9d4aaab683546c8f471448d5810796
I20260928 02:47:49.151144 16562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267: No bootstrap required, opened a new log
I20260928 02:47:49.152747 16562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260928 02:47:49.152902 16562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:49.152968 16562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c721d0548b64bd894c491948cd64267, State: Initialized, Role: FOLLOWER
I20260928 02:47:49.153108 16562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260928 02:47:49.153205 16562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:49.153249 16562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:49.153297 16562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:49.154132 16562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260928 02:47:49.154253 16562 leader_election.cc:302] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c721d0548b64bd894c491948cd64267; no voters:
I20260928 02:47:49.154421 16562 leader_election.cc:288] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:49.154450 16567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:49.154578 16567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [term 1 LEADER]: Becoming Leader. State: Replica: 2c721d0548b64bd894c491948cd64267, State: Running, Role: LEADER
I20260928 02:47:49.154696 16562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:49.154713 16567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } }
I20260928 02:47:49.155128 16567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c721d0548b64bd894c491948cd64267. Latest consensus state: current_term: 1 leader_uuid: "2c721d0548b64bd894c491948cd64267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260928 02:47:49.155275 16567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:49.155711 16576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:49.156250 16576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:49.157282 16568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c721d0548b64bd894c491948cd64267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c721d0548b64bd894c491948cd64267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35971 } } }
I20260928 02:47:49.157433 16568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:49.157675 16576 catalog_manager.cc:1347] Generated new cluster ID: e99b10b84035402eadb29099b6d1a41d
I20260928 02:47:49.157723 16576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:47:49 dist-test-slave-b5p5 krb5kdc[16481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:47:49.163393 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:49.151121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:54399 (local address 127.0.0.1:35971)
0928 02:47:49.151298 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:49.151309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:49.151340 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:47:49.152747 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:49.152754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:49.152805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:49.152967 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:49.153720 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:49.154910 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:49.156042 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:49.156663 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:49.156791 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:49.159563 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:49.159566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:49.159567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:49.159578 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:49.161468 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:49.161919 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:49.161921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:49.161922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:49.162039 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:49.162280 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:49.162281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:49.162281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:49.162572 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:49.163047 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:49.163119 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:47:49.163210 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260928 02:47:49.164305 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:49.167366 16576 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:49.168048 16576 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:49.172647 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:49.165754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:54385 (local address 127.0.0.1:35971)
0928 02:47:49.165788 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:49.165790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:49.165847 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:47:49.165904 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:49.165904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:49.165913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:49.165979 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:49.166559 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:49.167317 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:49.168270 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:49.168592 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:49.168664 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:49.169924 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:49.169925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:49.169925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:49.169932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:49.171492 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:49.171900 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:49.171901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:49.171901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:49.171957 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:49.172174 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:49.172175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:49.172175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:49.172295 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:49.172410 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:49.172506 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:47:49.172565 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:47:49.176146 16576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c721d0548b64bd894c491948cd64267: Generated new TSK 0
I20260928 02:47:49.176299 16576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:49.179936 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16494
W20260928 02:47:49.183871 16479 connection.cc:535] client connection to 127.0.0.1:35971 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35971): Connection reset by peer (error 104)
2026-09-28T02:47:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:49 dist-test-slave-b5p5 krb5kdc[16593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:49 dist-test-slave-b5p5 krb5kdc[16593](info): set up 2 sockets
Sep 28 02:47:49 dist-test-slave-b5p5 krb5kdc[16593](info): commencing operation
krb5kdc: starting...
W20260928 02:47:50.202569 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:50 dist-test-slave-b5p5 krb5kdc[16593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:50.312561 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:50.390408 16606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:50.390654 16606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:50.390691 16606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:50.391803 16606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:50.391877 16606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:50.391901 16606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:50.391929 16606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:50.391959 16606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:50.393409 16606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:50.393678 16606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:50.393890 16606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:50.398367 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:50.398419 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:50.398427 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:50.398643 16606 server_base.cc:1034] running on GCE node
I20260928 02:47:50.398927 16606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:50.399197 16606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:50.400373 16606 hybrid_clock.cc:648] HybridClock initialized: now 1790563670400356 us; error 28 us; skew 500 ppm
Sep 28 02:47:50 dist-test-slave-b5p5 krb5kdc[16593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:50.402667 16606 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:50.403082 16606 webserver.cc:460] Webserver started at http://127.0.0.1:33387/ using document root <none> and password file <none>
I20260928 02:47:50.403298 16606 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:50.403367 16606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:50.403496 16606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:50.404464 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "54a78e139267470aaefe06bc119af962"
format_stamp: "Formatted at 2026-09-28 02:47:50 on dist-test-slave-b5p5"
server_key: "d261912015837a8ad658d18e56842829"
server_key_iv: "48006f021740805a386004ca6a4c2a6e"
server_key_version: "encryptionkey@0"
I20260928 02:47:50.404812 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "54a78e139267470aaefe06bc119af962"
format_stamp: "Formatted at 2026-09-28 02:47:50 on dist-test-slave-b5p5"
server_key: "d261912015837a8ad658d18e56842829"
server_key_iv: "48006f021740805a386004ca6a4c2a6e"
server_key_version: "encryptionkey@0"
I20260928 02:47:50.406139 16606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:47:50.406890 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:50.407096 16606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:50.407191 16606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "54a78e139267470aaefe06bc119af962"
format_stamp: "Formatted at 2026-09-28 02:47:50 on dist-test-slave-b5p5"
server_key: "d261912015837a8ad658d18e56842829"
server_key_iv: "48006f021740805a386004ca6a4c2a6e"
server_key_version: "encryptionkey@0"
I20260928 02:47:50.407276 16606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:50.421613 16606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:50.423604 16606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:50.423759 16606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:50.427414 16606 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37285
I20260928 02:47:50.427459 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37285 every 8 connection(s)
I20260928 02:47:50.427759 16606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:50.428555 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:50.430689 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962: Bootstrap starting.
I20260928 02:47:50.431231 16674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:50.431506 16674 log.cc:824] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:50.432158 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962: No bootstrap required, opened a new log
I20260928 02:47:50.433655 16674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260928 02:47:50.433802 16674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:50.433847 16674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a78e139267470aaefe06bc119af962, State: Initialized, Role: FOLLOWER
I20260928 02:47:50.434028 16674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260928 02:47:50.434135 16674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:50.434178 16674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:50.434230 16674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:50.434789 16674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260928 02:47:50.434901 16674 leader_election.cc:302] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54a78e139267470aaefe06bc119af962; no voters:
I20260928 02:47:50.435072 16674 leader_election.cc:288] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:50.435098 16677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:50.435252 16677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [term 1 LEADER]: Becoming Leader. State: Replica: 54a78e139267470aaefe06bc119af962, State: Running, Role: LEADER
I20260928 02:47:50.435314 16674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:50.435382 16677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } }
I20260928 02:47:50.435760 16678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54a78e139267470aaefe06bc119af962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260928 02:47:50.435770 16679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a78e139267470aaefe06bc119af962. Latest consensus state: current_term: 1 leader_uuid: "54a78e139267470aaefe06bc119af962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a78e139267470aaefe06bc119af962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37285 } } }
I20260928 02:47:50.435930 16679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:50.435916 16678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:50.436292 16682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:50.436610 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16606
I20260928 02:47:50.436734 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:50.436831 16682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:50.436902 28086 external_mini_cluster.cc:1428] Setting key f84bbb0a3fa950a0fc72fba47cae0203
I20260928 02:47:50.438323 16682 catalog_manager.cc:1347] Generated new cluster ID: 6dfc18961aa04b96bc82c964c9f7c35c
I20260928 02:47:50.438393 16682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:50.443606 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:50.437339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:47381 (local address 127.0.0.1:37285)
0928 02:47:50.438368 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:50.438376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:50.438404 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:47:50.438528 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:50.438536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:50.438591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:50.438720 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:50.438951 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:50.440357 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:50.441391 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:50.442022 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:50.442128 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:50.442833 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:50.442837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:50.442837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:50.443046 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:50.443082 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:50.443096 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:50.443197 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:50.443281 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:47:50.443413 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":69,"threads_started":1}
I20260928 02:47:50.444519 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:50.449435 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:50.445993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:41773 (local address 127.0.0.1:37285)
0928 02:47:50.446035 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:50.446036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:50.446220 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:47:50.446263 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:50.446264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:50.446273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:50.446323 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:50.446541 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:50.447253 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:50.448220 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:50.448527 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:50.448587 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:50.449204 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:50.449205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:50.449205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:50.449209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:50.449221 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:50.449225 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:50.449282 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:50.449360 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:47:50.449370 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:47:50.455942 16682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:50.456406 16682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:50.460686 16682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54a78e139267470aaefe06bc119af962: Generated new TSK 0
I20260928 02:47:50.460872 16682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:50.466053 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16606
W20260928 02:47:50.470146 16591 connection.cc:535] client connection to 127.0.0.1:37285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37285): Connection reset by peer (error 104)
2026-09-28T02:47:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:50 dist-test-slave-b5p5 krb5kdc[16705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:50 dist-test-slave-b5p5 krb5kdc[16705](info): set up 2 sockets
Sep 28 02:47:50 dist-test-slave-b5p5 krb5kdc[16705](info): commencing operation
krb5kdc: starting...
W20260928 02:47:51.494380 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:51 dist-test-slave-b5p5 krb5kdc[16705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:51.604970 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:51.681928 16718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:51.682161 16718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:51.682202 16718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:51.683344 16718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:51.683413 16718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:51.683437 16718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:51.683472 16718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:51.683514 16718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:51.684924 16718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:51.685189 16718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:51.685403 16718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:51.690016 16724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:51.690027 16723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:51.690124 16718 server_base.cc:1034] running on GCE node
W20260928 02:47:51.690114 16726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:51.690562 16718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:51.690830 16718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:51.691965 16718 hybrid_clock.cc:648] HybridClock initialized: now 1790563671691952 us; error 30 us; skew 500 ppm
Sep 28 02:47:51 dist-test-slave-b5p5 krb5kdc[16705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:51.694192 16718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:51.694648 16718 webserver.cc:460] Webserver started at http://127.0.0.1:38749/ using document root <none> and password file <none>
I20260928 02:47:51.694860 16718 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:51.694926 16718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:51.695056 16718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:51.696029 16718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "ff6fbdfcd15246e4824e2b96c0d13754"
format_stamp: "Formatted at 2026-09-28 02:47:51 on dist-test-slave-b5p5"
server_key: "a42bfeea1900bcbbf6474117101aa437"
server_key_iv: "2d06b53be0727479172d0a9674bc504e"
server_key_version: "encryptionkey@0"
I20260928 02:47:51.696376 16718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "ff6fbdfcd15246e4824e2b96c0d13754"
format_stamp: "Formatted at 2026-09-28 02:47:51 on dist-test-slave-b5p5"
server_key: "a42bfeea1900bcbbf6474117101aa437"
server_key_iv: "2d06b53be0727479172d0a9674bc504e"
server_key_version: "encryptionkey@0"
I20260928 02:47:51.697643 16718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:51.698297 16733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:51.698511 16718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:51.698601 16718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "ff6fbdfcd15246e4824e2b96c0d13754"
format_stamp: "Formatted at 2026-09-28 02:47:51 on dist-test-slave-b5p5"
server_key: "a42bfeea1900bcbbf6474117101aa437"
server_key_iv: "2d06b53be0727479172d0a9674bc504e"
server_key_version: "encryptionkey@0"
I20260928 02:47:51.698686 16718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:51.701788 16718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:51.703544 16718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:51.703703 16718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:51.707399 16718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43903
I20260928 02:47:51.707438 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43903 every 8 connection(s)
I20260928 02:47:51.707782 16718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:51.708734 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16718
I20260928 02:47:51.708767 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:51.708878 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:51.709054 28086 external_mini_cluster.cc:1428] Setting key 8e01d4c0332a9691dc6d6b3d3a308e1d
I20260928 02:47:51.711521 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:51.709560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:60885 (local address 127.0.0.1:43903)
0928 02:47:51.709751 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:51.709760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:51.709791 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:47:51.710289 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:51.710298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:51.710333 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:51.710522 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:51.711119 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:51.711122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:51.711123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:51.711127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:51.711167 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:51.711180 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:51.711257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:51.711308 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:47:51.711349 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 02:47:51.712072 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754: Bootstrap starting.
I20260928 02:47:51.712495 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:51.712756 16786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:51.713029 16786 log.cc:824] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:51.713722 16786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754: No bootstrap required, opened a new log
I20260928 02:47:51.714924 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:51.713858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:39009 (local address 127.0.0.1:43903)
0928 02:47:51.713917 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:51.713919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:51.714003 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:47:51.714080 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:51.714081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:51.714085 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:51.714141 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:51.714722 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:51.714723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:51.714723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:51.714728 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:51.714742 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:51.714746 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:51.714813 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:51.714848 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:47:51.714863 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:47:51.715991 16786 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } }
I20260928 02:47:51.716113 16786 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:51.716135 16786 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6fbdfcd15246e4824e2b96c0d13754, State: Initialized, Role: FOLLOWER
I20260928 02:47:51.716243 16786 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } }
I20260928 02:47:51.716331 16786 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:51.716353 16786 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:51.716380 16786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:51.716979 16786 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } }
I20260928 02:47:51.717062 16786 leader_election.cc:302] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6fbdfcd15246e4824e2b96c0d13754; no voters:
I20260928 02:47:51.717170 16786 leader_election.cc:288] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:51.717234 16796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:51.717391 16786 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:51.717407 16796 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [term 1 LEADER]: Becoming Leader. State: Replica: ff6fbdfcd15246e4824e2b96c0d13754, State: Running, Role: LEADER
I20260928 02:47:51.717600 16796 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } }
I20260928 02:47:51.717967 16797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } } }
I20260928 02:47:51.717974 16798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6fbdfcd15246e4824e2b96c0d13754. Latest consensus state: current_term: 1 leader_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fbdfcd15246e4824e2b96c0d13754" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43903 } } }
I20260928 02:47:51.718096 16797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:51.718112 16798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:51.718331 16805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:51.718850 16805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:51.720300 16805 catalog_manager.cc:1347] Generated new cluster ID: 82001a63a6e54bdf86a032cd16579391
I20260928 02:47:51.720359 16805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:51.735637 16805 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:51.735816 16805 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:51.737771 16805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6fbdfcd15246e4824e2b96c0d13754: Generated new TSK 0
I20260928 02:47:51.737946 16805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:51.739866 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16718
2026-09-28T02:47:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:51 dist-test-slave-b5p5 krb5kdc[16817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:51 dist-test-slave-b5p5 krb5kdc[16817](info): set up 2 sockets
Sep 28 02:47:51 dist-test-slave-b5p5 krb5kdc[16817](info): commencing operation
krb5kdc: starting...
W20260928 02:47:52.766456 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:52 dist-test-slave-b5p5 krb5kdc[16817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:52.877046 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:52.955026 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:52.955215 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:52.955236 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:52.956355 16830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:52.956396 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:52.956411 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:52.956456 16830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:52.956482 16830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:52.957875 16830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:52.958110 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:52.958321 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:52.962870 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:52.962881 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:52.962924 16830 server_base.cc:1034] running on GCE node
W20260928 02:47:52.962862 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:52.963351 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:52.963577 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:52.964715 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790563672964695 us; error 39 us; skew 500 ppm
Sep 28 02:47:52 dist-test-slave-b5p5 krb5kdc[16817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:52.967021 16830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:52.967366 16830 webserver.cc:460] Webserver started at http://127.0.0.1:46333/ using document root <none> and password file <none>
I20260928 02:47:52.967535 16830 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:52.967573 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:52.967657 16830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:52.968561 16830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "9b58a0b3cb794f4fb92121e06160fb1b"
format_stamp: "Formatted at 2026-09-28 02:47:52 on dist-test-slave-b5p5"
server_key: "ba49e8850afbf7459f46b9a3557a2f1e"
server_key_iv: "daecfbbab3cc23273f928be033745334"
server_key_version: "encryptionkey@0"
I20260928 02:47:52.968838 16830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "9b58a0b3cb794f4fb92121e06160fb1b"
format_stamp: "Formatted at 2026-09-28 02:47:52 on dist-test-slave-b5p5"
server_key: "ba49e8850afbf7459f46b9a3557a2f1e"
server_key_iv: "daecfbbab3cc23273f928be033745334"
server_key_version: "encryptionkey@0"
I20260928 02:47:52.970009 16830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:47:52.970688 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:52.970855 16830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:52.970940 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "9b58a0b3cb794f4fb92121e06160fb1b"
format_stamp: "Formatted at 2026-09-28 02:47:52 on dist-test-slave-b5p5"
server_key: "ba49e8850afbf7459f46b9a3557a2f1e"
server_key_iv: "daecfbbab3cc23273f928be033745334"
server_key_version: "encryptionkey@0"
I20260928 02:47:52.970990 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:52.979861 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:52.981527 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:52.981627 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:52.985298 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34133
I20260928 02:47:52.985364 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34133 every 8 connection(s)
I20260928 02:47:52.985661 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:52.986429 16898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:52.988466 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: Bootstrap starting.
I20260928 02:47:52.989015 16898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:52.989289 16898 log.cc:824] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:52.989929 16898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: No bootstrap required, opened a new log
I20260928 02:47:52.990890 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16830
I20260928 02:47:52.990979 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:52.991191 28086 external_mini_cluster.cc:1428] Setting key 9063c2af20d1dd6fb56c93897f500534
I20260928 02:47:52.991792 16898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260928 02:47:52.991948 16898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:52.991993 16898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b58a0b3cb794f4fb92121e06160fb1b, State: Initialized, Role: FOLLOWER
I20260928 02:47:52.992125 16898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260928 02:47:52.992201 16898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:52.992259 16898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:52.992309 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:52.993108 16898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260928 02:47:52.993224 16898 leader_election.cc:302] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b58a0b3cb794f4fb92121e06160fb1b; no voters:
I20260928 02:47:52.993371 16898 leader_election.cc:288] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:52.993392 16903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:52.993500 16903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [term 1 LEADER]: Becoming Leader. State: Replica: 9b58a0b3cb794f4fb92121e06160fb1b, State: Running, Role: LEADER
I20260928 02:47:52.993623 16898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:52.993636 16903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } }
I20260928 02:47:52.994892 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b58a0b3cb794f4fb92121e06160fb1b. Latest consensus state: current_term: 1 leader_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } } }
I20260928 02:47:52.994946 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b58a0b3cb794f4fb92121e06160fb1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34133 } } }
I20260928 02:47:52.994999 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:52.995002 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b [sys.catalog]: This master's current role is: LEADER
W20260928 02:47:52.995517 16918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:47:52.995568 16918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:47:52.995667 16919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:52.995836 16919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:52.997398 16919 catalog_manager.cc:1347] Generated new cluster ID: 24bb68355ed2447695c2f463342130e3
I20260928 02:47:52.997452 16919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:47:52 dist-test-slave-b5p5 krb5kdc[16817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:47:53.001893 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:52.991645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:60459 (local address 127.0.0.1:34133)
0928 02:47:52.991842 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:52.991851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:52.991917 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:47:52.992032 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:52.992039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:52.992094 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:52.992243 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:52.992862 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:52.994155 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:52.995339 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:52.995970 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:52.996069 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:52.998771 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:52.998774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:52.998775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:52.998786 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:53.000395 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:53.000786 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:53.000789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:53.000790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:53.000894 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:53.001131 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:53.001132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:53.001132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:53.001347 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:53.001535 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:53.001614 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:47:53.001714 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260928 02:47:53.002821 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:53.012477 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:53.004128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:55233 (local address 127.0.0.1:34133)
0928 02:47:53.004163 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:53.004165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:53.006663 (+ 2498us) server_negotiation.cc:407] Connection header received
0928 02:47:53.006742 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:53.006743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:53.006751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:53.006795 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:53.007281 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:53.007747 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:53.008427 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:53.008652 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:53.008734 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:53.009850 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:53.009851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:53.009852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:53.009859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:53.011330 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:53.011688 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:53.011689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:53.011689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:53.011740 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:53.011972 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:53.011973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:53.011973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:53.012079 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:53.012193 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:53.012310 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:47:53.012370 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:47:53.017602 16919 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:53.018260 16919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:53.031486 16919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b58a0b3cb794f4fb92121e06160fb1b: Generated new TSK 0
I20260928 02:47:53.031647 16919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:53.039541 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16830
W20260928 02:47:53.043773 16815 connection.cc:535] client connection to 127.0.0.1:34133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34133): Connection reset by peer (error 104)
2026-09-28T02:47:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:53 dist-test-slave-b5p5 krb5kdc[16929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:53 dist-test-slave-b5p5 krb5kdc[16929](info): set up 2 sockets
Sep 28 02:47:53 dist-test-slave-b5p5 krb5kdc[16929](info): commencing operation
krb5kdc: starting...
W20260928 02:47:54.078552 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:54 dist-test-slave-b5p5 krb5kdc[16929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:54.188964 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:54.266579 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:54.266796 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:54.266835 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:54.267966 16942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:54.268035 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:54.268061 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:54.268093 16942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:54.268119 16942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:54.269559 16942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:54.269819 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:54.270042 16942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:54.274845 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:54.274868 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:54.274863 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:54.275098 16942 server_base.cc:1034] running on GCE node
I20260928 02:47:54.275336 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:54.275615 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:54.276759 16942 hybrid_clock.cc:648] HybridClock initialized: now 1790563674276741 us; error 37 us; skew 500 ppm
Sep 28 02:47:54 dist-test-slave-b5p5 krb5kdc[16929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:54.281930 16942 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:54.282364 16942 webserver.cc:460] Webserver started at http://127.0.0.1:40601/ using document root <none> and password file <none>
I20260928 02:47:54.282629 16942 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:54.282673 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:54.282757 16942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:54.283705 16942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "be177c449626437d8bb618ac5874745d"
format_stamp: "Formatted at 2026-09-28 02:47:54 on dist-test-slave-b5p5"
server_key: "ed5647a7baa2973fe2aedc5342c8768d"
server_key_iv: "46d86a38cbbb40aba81675e7b3b059cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:54.283988 16942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "be177c449626437d8bb618ac5874745d"
format_stamp: "Formatted at 2026-09-28 02:47:54 on dist-test-slave-b5p5"
server_key: "ed5647a7baa2973fe2aedc5342c8768d"
server_key_iv: "46d86a38cbbb40aba81675e7b3b059cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:54.285171 16942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:47:54.285806 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:54.285975 16942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:54.286085 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "be177c449626437d8bb618ac5874745d"
format_stamp: "Formatted at 2026-09-28 02:47:54 on dist-test-slave-b5p5"
server_key: "ed5647a7baa2973fe2aedc5342c8768d"
server_key_iv: "46d86a38cbbb40aba81675e7b3b059cb"
server_key_version: "encryptionkey@0"
I20260928 02:47:54.286159 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:54.299532 16942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:54.301808 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:54.301962 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:54.305639 16942 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41249
I20260928 02:47:54.305687 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41249 every 8 connection(s)
I20260928 02:47:54.306043 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:54.306950 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:54.309069 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d: Bootstrap starting.
I20260928 02:47:54.309618 17010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:54.309890 17010 log.cc:824] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:54.310587 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d: No bootstrap required, opened a new log
I20260928 02:47:54.312091 17010 raft_consensus.cc:348] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260928 02:47:54.312223 17010 raft_consensus.cc:374] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:54.312244 17010 raft_consensus.cc:729] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be177c449626437d8bb618ac5874745d, State: Initialized, Role: FOLLOWER
I20260928 02:47:54.312405 17010 consensus_queue.cc:260] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260928 02:47:54.312489 17010 raft_consensus.cc:388] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:54.312547 17010 raft_consensus.cc:482] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:54.312599 17010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:54.312971 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 16942
I20260928 02:47:54.313063 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:54.313154 17010 raft_consensus.cc:504] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260928 02:47:54.313272 17010 leader_election.cc:302] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be177c449626437d8bb618ac5874745d; no voters:
I20260928 02:47:54.313308 28086 external_mini_cluster.cc:1428] Setting key c77c6d8d9088bd15c884f67968e25ca7
I20260928 02:47:54.313441 17010 leader_election.cc:288] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:54.313477 17013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:54.313608 17013 raft_consensus.cc:686] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [term 1 LEADER]: Becoming Leader. State: Replica: be177c449626437d8bb618ac5874745d, State: Running, Role: LEADER
I20260928 02:47:54.313777 17013 consensus_queue.cc:237] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } }
I20260928 02:47:54.313908 17010 sys_catalog.cc:564] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:54.314113 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be177c449626437d8bb618ac5874745d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } } }
I20260928 02:47:54.314180 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:54.314283 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [sys.catalog]: SysCatalogTable state changed. Reason: New leader be177c449626437d8bb618ac5874745d. Latest consensus state: current_term: 1 leader_uuid: "be177c449626437d8bb618ac5874745d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be177c449626437d8bb618ac5874745d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41249 } } }
I20260928 02:47:54.314409 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:54.314533 17019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:54.315089 17019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:54.317535 17019 catalog_manager.cc:1347] Generated new cluster ID: b44dcbadf9b84e01a3409085af3b0a05
I20260928 02:47:54.317591 17019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:54.319718 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:54.313710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:50975 (local address 127.0.0.1:41249)
0928 02:47:54.315396 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:54.315406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:54.315438 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:47:54.315556 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:54.315561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:54.315603 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:54.315743 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:54.315975 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:54.316924 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:54.317904 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:54.318309 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:54.318441 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:54.319129 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:54.319133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:54.319133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:54.319303 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:54.319330 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:54.319340 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:54.319433 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:54.319511 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:47:54.319545 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1637,"thread_start_us":59,"threads_started":1}
I20260928 02:47:54.320592 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:54.324738 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:54.321937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:45829 (local address 127.0.0.1:41249)
0928 02:47:54.322011 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:54.322013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:54.322067 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:47:54.322137 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:54.322137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:54.322147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:54.322198 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:54.322340 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:54.322910 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:54.323610 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:54.323826 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:54.323901 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:54.324517 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:54.324519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:54.324519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:54.324531 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:54.324543 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:54.324547 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:54.324599 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:54.324658 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:47:54.324669 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:47:54.341660 17019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:54.342156 17019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:54.344880 17019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be177c449626437d8bb618ac5874745d: Generated new TSK 0
I20260928 02:47:54.345044 17019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:54.347775 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 16942
W20260928 02:47:54.352274 16927 connection.cc:535] client connection to 127.0.0.1:41249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41249): Connection reset by peer (error 104)
2026-09-28T02:47:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:54 dist-test-slave-b5p5 krb5kdc[17041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:54 dist-test-slave-b5p5 krb5kdc[17041](info): set up 2 sockets
Sep 28 02:47:54 dist-test-slave-b5p5 krb5kdc[17041](info): commencing operation
krb5kdc: starting...
W20260928 02:47:55.386569 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:55 dist-test-slave-b5p5 krb5kdc[17041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:47:55.500655 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:55.580332 17054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:55.580515 17054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:55.580538 17054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:55.581581 17054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:55.581614 17054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:55.581627 17054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:55.581640 17054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:55.581650 17054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:55.583127 17054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:55.583302 17054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:55.583458 17054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:47:55.588109 17054 server_base.cc:1034] running on GCE node
W20260928 02:47:55.588093 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:55.588116 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:55.588094 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:55.588553 17054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:55.588778 17054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:55.589916 17054 hybrid_clock.cc:648] HybridClock initialized: now 1790563675589906 us; error 28 us; skew 500 ppm
Sep 28 02:47:55 dist-test-slave-b5p5 krb5kdc[17041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:55.592350 17054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:47:55.592749 17054 webserver.cc:460] Webserver started at http://127.0.0.1:32899/ using document root <none> and password file <none>
I20260928 02:47:55.592980 17054 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:55.593046 17054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:55.593189 17054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:55.594195 17054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "373b45b2c08c4ba19adf84d60e9fe7ed"
format_stamp: "Formatted at 2026-09-28 02:47:55 on dist-test-slave-b5p5"
server_key: "67cd710a2ef797a7aedeb22fa9b62aba"
server_key_iv: "3fffcc2c394a3e402d63b30116d43cda"
server_key_version: "encryptionkey@0"
I20260928 02:47:55.594605 17054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "373b45b2c08c4ba19adf84d60e9fe7ed"
format_stamp: "Formatted at 2026-09-28 02:47:55 on dist-test-slave-b5p5"
server_key: "67cd710a2ef797a7aedeb22fa9b62aba"
server_key_iv: "3fffcc2c394a3e402d63b30116d43cda"
server_key_version: "encryptionkey@0"
I20260928 02:47:55.595856 17054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:47:55.596522 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:55.596678 17054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:55.596776 17054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "373b45b2c08c4ba19adf84d60e9fe7ed"
format_stamp: "Formatted at 2026-09-28 02:47:55 on dist-test-slave-b5p5"
server_key: "67cd710a2ef797a7aedeb22fa9b62aba"
server_key_iv: "3fffcc2c394a3e402d63b30116d43cda"
server_key_version: "encryptionkey@0"
I20260928 02:47:55.596868 17054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:55.600041 17054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:55.602190 17054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:55.602356 17054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:55.606045 17054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36489
I20260928 02:47:55.606084 17121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36489 every 8 connection(s)
I20260928 02:47:55.606400 17054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:55.607398 17122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:55.609411 17122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed: Bootstrap starting.
I20260928 02:47:55.609947 17122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:55.610221 17122 log.cc:824] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:55.610998 17122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed: No bootstrap required, opened a new log
I20260928 02:47:55.612515 17122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260928 02:47:55.612641 17122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:55.612663 17122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373b45b2c08c4ba19adf84d60e9fe7ed, State: Initialized, Role: FOLLOWER
I20260928 02:47:55.612838 17122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260928 02:47:55.612920 17122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:55.612973 17122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:55.613020 17122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:55.613545 17122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260928 02:47:55.613651 17122 leader_election.cc:302] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 373b45b2c08c4ba19adf84d60e9fe7ed; no voters:
I20260928 02:47:55.613828 17122 leader_election.cc:288] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:55.613907 17125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:55.614066 17122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:55.614084 17125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [term 1 LEADER]: Becoming Leader. State: Replica: 373b45b2c08c4ba19adf84d60e9fe7ed, State: Running, Role: LEADER
I20260928 02:47:55.614166 17125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } }
I20260928 02:47:55.614512 17126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } } }
I20260928 02:47:55.614578 17127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 373b45b2c08c4ba19adf84d60e9fe7ed. Latest consensus state: current_term: 1 leader_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "373b45b2c08c4ba19adf84d60e9fe7ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36489 } } }
I20260928 02:47:55.614677 17127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:55.614663 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17054
I20260928 02:47:55.614609 17126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:55.614770 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:55.614956 17129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:55.615005 28086 external_mini_cluster.cc:1428] Setting key 4de75b2004ddbd8d84f49805839c0090
I20260928 02:47:55.615605 17129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:55.617456 17129 catalog_manager.cc:1347] Generated new cluster ID: 7440df22ac324d34b6638c2dfd338c53
I20260928 02:47:55.617506 17129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:55.617816 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:55.615617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:46457 (local address 127.0.0.1:36489)
0928 02:47:55.616332 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:55.616340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:55.616369 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:47:55.616483 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:55.616490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:55.616532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:55.616700 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:55.617370 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:55.617374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:55.617374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:55.617378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:55.617429 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:55.617444 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:55.617482 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:55.617619 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:47:55.617660 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":81,"threads_started":1}
I20260928 02:47:55.618875 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:55.621320 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:55.620284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.203:56655 (local address 127.0.0.1:36489)
0928 02:47:55.620314 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:55.620315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:55.620457 (+ 142us) server_negotiation.cc:407] Connection header received
0928 02:47:55.620486 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:55.620487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:55.620492 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:55.620556 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:55.621113 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:55.621114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:55.621114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:55.621118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:55.621132 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:55.621136 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:55.621199 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:55.621247 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:47:55.621261 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:47:55.636173 17129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:55.636322 17129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:55.641427 17129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 373b45b2c08c4ba19adf84d60e9fe7ed: Generated new TSK 0
I20260928 02:47:55.641614 17129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:55.647572 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17054
2026-09-28T02:47:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:55 dist-test-slave-b5p5 krb5kdc[17153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:55 dist-test-slave-b5p5 krb5kdc[17153](info): set up 2 sockets
Sep 28 02:47:55 dist-test-slave-b5p5 krb5kdc[17153](info): commencing operation
krb5kdc: starting...
W20260928 02:47:56.678589 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:56.789707 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37227
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:56.867187 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:56.867390 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:56.867427 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:56.868502 17166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:56.868566 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:56.868589 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:56.868616 17166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:56.868673 17166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:56.870092 17166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:56.870347 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:56.870568 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:56.875022 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:56.875061 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:56.875237 17166 server_base.cc:1034] running on GCE node
W20260928 02:47:56.875254 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:56.875535 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:56.875805 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:56.876951 17166 hybrid_clock.cc:648] HybridClock initialized: now 1790563676876937 us; error 26 us; skew 500 ppm
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:56.879212 17166 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:56.879653 17166 webserver.cc:460] Webserver started at http://127.27.109.190:36179/ using document root <none> and password file <none>
I20260928 02:47:56.879868 17166 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:56.879931 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:56.880080 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:56.881044 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "24f436eeaa384678a81412da973d1a92"
format_stamp: "Formatted at 2026-09-28 02:47:56 on dist-test-slave-b5p5"
server_key: "1e9bfc29663a43185b90019dc221276b"
server_key_iv: "e1ed2a8bc8e339c7e778b223e8abbf7b"
server_key_version: "encryptionkey@0"
I20260928 02:47:56.881397 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "24f436eeaa384678a81412da973d1a92"
format_stamp: "Formatted at 2026-09-28 02:47:56 on dist-test-slave-b5p5"
server_key: "1e9bfc29663a43185b90019dc221276b"
server_key_iv: "e1ed2a8bc8e339c7e778b223e8abbf7b"
server_key_version: "encryptionkey@0"
I20260928 02:47:56.882722 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:56.883500 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:56.883675 17166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:47:56.883772 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "24f436eeaa384678a81412da973d1a92"
format_stamp: "Formatted at 2026-09-28 02:47:56 on dist-test-slave-b5p5"
server_key: "1e9bfc29663a43185b90019dc221276b"
server_key_iv: "e1ed2a8bc8e339c7e778b223e8abbf7b"
server_key_version: "encryptionkey@0"
I20260928 02:47:56.883847 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:56.906139 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:56.908260 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:56.908392 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:56.912256 17166 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37227
I20260928 02:47:56.912308 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37227 every 8 connection(s)
I20260928 02:47:56.912590 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:56.913381 17234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:56.913715 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17166
I20260928 02:47:56.913801 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:56.913966 28086 external_mini_cluster.cc:1428] Setting key 34b1d6034c10693271ba2bb7e80b0d41
I20260928 02:47:56.916687 17234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92: Bootstrap starting.
I20260928 02:47:56.917306 17234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:56.917600 17234 log.cc:824] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:56.918232 17234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92: No bootstrap required, opened a new log
I20260928 02:47:56.920181 17234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } }
I20260928 02:47:56.920327 17234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:56.920375 17234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f436eeaa384678a81412da973d1a92, State: Initialized, Role: FOLLOWER
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:47:56.920527 17234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } }
I20260928 02:47:56.920614 17234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:56.920648 17234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:56.920703 17234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:56.921304 17234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } }
I20260928 02:47:56.921415 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f436eeaa384678a81412da973d1a92; no voters:
I20260928 02:47:56.921588 17234 leader_election.cc:288] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:56.921645 17239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:56.921787 17239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [term 1 LEADER]: Becoming Leader. State: Replica: 24f436eeaa384678a81412da973d1a92, State: Running, Role: LEADER
I20260928 02:47:56.921844 17234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:56.921927 17239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } }
I20260928 02:47:56.922309 17240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f436eeaa384678a81412da973d1a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } } }
I20260928 02:47:56.922482 17240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:56.922900 17248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:56.923305 17241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f436eeaa384678a81412da973d1a92. Latest consensus state: current_term: 1 leader_uuid: "24f436eeaa384678a81412da973d1a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f436eeaa384678a81412da973d1a92" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37227 } } }
I20260928 02:47:56.923460 17248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:56.923712 17241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92 [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:56.924724 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:56.914414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48095 (local address 127.27.109.190:37227)
0928 02:47:56.914610 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:56.914619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:56.914652 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:47:56.915127 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:56.915136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:56.915194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:56.915339 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:56.915905 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:56.917249 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:56.918037 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:56.918536 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:56.918642 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:56.920988 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:56.920991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:56.920992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:56.921001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:56.923010 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:56.923488 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:56.923491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:56.923492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:56.923617 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:56.923829 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:56.923830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:56.923830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:56.924059 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:56.924236 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:56.924509 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:47:56.924580 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260928 02:47:56.925654 17248 catalog_manager.cc:1347] Generated new cluster ID: e2b857d1994e4ef0bd8b3ac27b9d0011
I20260928 02:47:56.925699 17248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:56.925701 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:56.932910 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:56.926957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34735 (local address 127.27.109.190:37227)
0928 02:47:56.926990 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:56.926992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:56.927110 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:47:56.927186 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:56.927187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:56.927196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:56.927249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:56.927733 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:56.928181 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:56.928868 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:56.929101 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:56.929189 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:56.930298 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:56.930299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:56.930300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:47:56.930307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:56.931771 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:56.932127 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:56.932128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:56.932129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:56.932180 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:47:56.932408 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:47:56.932409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:47:56.932410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:47:56.932513 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:56.932617 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:56.932744 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:47:56.932804 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:47:56.933950 17248 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:56.934407 17248 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:56.937496 17248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f436eeaa384678a81412da973d1a92: Generated new TSK 0
I20260928 02:47:56.937645 17248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:56.940160 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17166
W20260928 02:47:56.943830 17151 connection.cc:535] client connection to 127.27.109.190:37227 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.190:37227): Connection reset by peer (error 104)
2026-09-28T02:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17265](info): set up 2 sockets
Sep 28 02:47:56 dist-test-slave-b5p5 krb5kdc[17265](info): commencing operation
krb5kdc: starting...
W20260928 02:47:57.970379 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:58 dist-test-slave-b5p5 krb5kdc[17265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:58.081745 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35531
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:58.159415 17278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:58.159605 17278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:58.159626 17278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:58.160816 17278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:58.160857 17278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:58.160872 17278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:58.160884 17278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:58.160933 17278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:58.162328 17278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:35531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:58.162591 17278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:58.162814 17278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:58.167487 17283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:58.167543 17286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:58.167559 17284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:58.167639 17278 server_base.cc:1034] running on GCE node
I20260928 02:47:58.167958 17278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:58.168229 17278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:58.169394 17278 hybrid_clock.cc:648] HybridClock initialized: now 1790563678169391 us; error 36 us; skew 500 ppm
Sep 28 02:47:58 dist-test-slave-b5p5 krb5kdc[17265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563678, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:58.171703 17278 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:58.172142 17278 webserver.cc:460] Webserver started at http://127.27.109.190:37193/ using document root <none> and password file <none>
I20260928 02:47:58.172355 17278 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:58.172421 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:58.172566 17278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:58.173525 17278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "24bd88a0c22e4fcbacd71309a13e772b"
format_stamp: "Formatted at 2026-09-28 02:47:58 on dist-test-slave-b5p5"
server_key: "62263256085b31d1bd5532c50bed2495"
server_key_iv: "d37991d0f6f10a62b34b0a383e74b05a"
server_key_version: "encryptionkey@0"
I20260928 02:47:58.173870 17278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "24bd88a0c22e4fcbacd71309a13e772b"
format_stamp: "Formatted at 2026-09-28 02:47:58 on dist-test-slave-b5p5"
server_key: "62263256085b31d1bd5532c50bed2495"
server_key_iv: "d37991d0f6f10a62b34b0a383e74b05a"
server_key_version: "encryptionkey@0"
I20260928 02:47:58.175243 17278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:47:58.175943 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:58.176110 17278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:47:58.176218 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "24bd88a0c22e4fcbacd71309a13e772b"
format_stamp: "Formatted at 2026-09-28 02:47:58 on dist-test-slave-b5p5"
server_key: "62263256085b31d1bd5532c50bed2495"
server_key_iv: "d37991d0f6f10a62b34b0a383e74b05a"
server_key_version: "encryptionkey@0"
I20260928 02:47:58.176291 17278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:58.188596 17278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:58.190217 17278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:58.190367 17278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:58.194124 17278 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:35531
I20260928 02:47:58.194155 17345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:35531 every 8 connection(s)
I20260928 02:47:58.194555 17278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:58.195310 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:58.195585 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17278
I20260928 02:47:58.195735 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:58.195904 28086 external_mini_cluster.cc:1428] Setting key 480c187c22711bfb977f18ef21c70ebf
I20260928 02:47:58.198292 17346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b: Bootstrap starting.
I20260928 02:47:58.198962 17346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:58.199242 17346 log.cc:824] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:58.199932 17346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b: No bootstrap required, opened a new log
I20260928 02:47:58.201856 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:58.196384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32777 (local address 127.27.109.190:35531)
0928 02:47:58.196575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:58.196583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:58.196618 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:47:58.197151 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:58.197159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:58.197216 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:58.197356 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:58.197514 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:58.198898 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:58.199874 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:58.200454 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:58.200562 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:58.201187 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:58.201191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:58.201191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:58.201390 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:58.201426 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:58.201439 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:58.201523 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:58.201599 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:47:58.201626 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 02:47:58.202409 17346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } }
I20260928 02:47:58.202600 17346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:58.202663 17346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24bd88a0c22e4fcbacd71309a13e772b, State: Initialized, Role: FOLLOWER
I20260928 02:47:58.202781 17346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } }
I20260928 02:47:58.202862 17346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:58.202919 17346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:58.202907 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:58.202971 17346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:58.203452 17346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } }
I20260928 02:47:58.203563 17346 leader_election.cc:302] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24bd88a0c22e4fcbacd71309a13e772b; no voters:
I20260928 02:47:58.203717 17346 leader_election.cc:288] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:58.203745 17352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:58.203922 17352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [term 1 LEADER]: Becoming Leader. State: Replica: 24bd88a0c22e4fcbacd71309a13e772b, State: Running, Role: LEADER
I20260928 02:47:58.204000 17346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:58.204114 17352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } }
I20260928 02:47:58.204541 17357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24bd88a0c22e4fcbacd71309a13e772b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } } }
I20260928 02:47:58.204684 17357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:58.204859 17357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bd88a0c22e4fcbacd71309a13e772b. Latest consensus state: current_term: 1 leader_uuid: "24bd88a0c22e4fcbacd71309a13e772b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bd88a0c22e4fcbacd71309a13e772b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 35531 } } }
I20260928 02:47:58.204926 17357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:58.205278 17365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:58.205855 17365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:58.207471 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:58.204217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57609 (local address 127.27.109.190:35531)
0928 02:47:58.204257 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:58.204258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:58.204306 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:47:58.204379 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:58.204380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:58.204389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:58.204436 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:58.204583 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:58.205302 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:58.206165 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:47:58.206392 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:47:58.206501 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:47:58.207120 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:58.207121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:58.207121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:58.207126 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:58.207141 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:58.207146 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:58.207198 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:58.207392 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:47:58.207403 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:47:58.207520 17365 catalog_manager.cc:1347] Generated new cluster ID: cf60ae209f8040978a9599f76bc701e4
I20260928 02:47:58.207556 17365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:58.221836 17365 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:58.222523 17365 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:58.227778 17365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24bd88a0c22e4fcbacd71309a13e772b: Generated new TSK 0
I20260928 02:47:58.227952 17365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:58.232810 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17278
W20260928 02:47:58.236876 17263 connection.cc:535] client connection to 127.27.109.190:35531 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.190:35531): Connection reset by peer (error 104)
2026-09-28T02:47:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:58 dist-test-slave-b5p5 krb5kdc[17377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:58 dist-test-slave-b5p5 krb5kdc[17377](info): set up 2 sockets
Sep 28 02:47:58 dist-test-slave-b5p5 krb5kdc[17377](info): commencing operation
krb5kdc: starting...
W20260928 02:47:59.246423 28086 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:47:59 dist-test-slave-b5p5 krb5kdc[17377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:47:59.356964 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:44605
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:44605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:47:59.438977 17390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:47:59.439153 17390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:47:59.439175 17390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:47:59.440358 17390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:47:59.440397 17390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:47:59.440411 17390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:47:59.440454 17390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:47:59.440488 17390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:47:59.441946 17390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:44605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:44605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:47:59.442157 17390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:47:59.442368 17390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:47:59.446928 17395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:59.446928 17398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:47:59.446944 17396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:47:59.447067 17390 server_base.cc:1034] running on GCE node
I20260928 02:47:59.447366 17390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:47:59.447633 17390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:47:59.448822 17390 hybrid_clock.cc:648] HybridClock initialized: now 1790563679448811 us; error 45 us; skew 500 ppm
Sep 28 02:47:59 dist-test-slave-b5p5 krb5kdc[17377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563679, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:47:59.456604 17390 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:47:59.457038 17390 webserver.cc:460] Webserver started at http://127.27.109.190:44325/ using document root <none> and password file <none>
I20260928 02:47:59.457260 17390 fs_manager.cc:359] Metadata directory not provided
I20260928 02:47:59.457329 17390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:47:59.457477 17390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:47:59.458600 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "97b9e2eceaa9452292c4fd1f0c9896ee"
format_stamp: "Formatted at 2026-09-28 02:47:59 on dist-test-slave-b5p5"
server_key: "e1ef25d84556915c8aeec1bc80c600d1"
server_key_iv: "c970e05916a7011aaf378d9cbdd0500b"
server_key_version: "encryptionkey@0"
I20260928 02:47:59.458953 17390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "97b9e2eceaa9452292c4fd1f0c9896ee"
format_stamp: "Formatted at 2026-09-28 02:47:59 on dist-test-slave-b5p5"
server_key: "e1ef25d84556915c8aeec1bc80c600d1"
server_key_iv: "c970e05916a7011aaf378d9cbdd0500b"
server_key_version: "encryptionkey@0"
I20260928 02:47:59.460235 17390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:47:59.460937 17405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:47:59.461119 17390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:47:59.461221 17390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "97b9e2eceaa9452292c4fd1f0c9896ee"
format_stamp: "Formatted at 2026-09-28 02:47:59 on dist-test-slave-b5p5"
server_key: "e1ef25d84556915c8aeec1bc80c600d1"
server_key_iv: "c970e05916a7011aaf378d9cbdd0500b"
server_key_version: "encryptionkey@0"
I20260928 02:47:59.461294 17390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:47:59.464713 17390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:47:59.467088 17390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:47:59.467239 17390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:47:59.471029 17390 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:44605
I20260928 02:47:59.471060 17457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:44605 every 8 connection(s)
I20260928 02:47:59.471371 17390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:47:59.472272 17458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:47:59.474396 17458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee: Bootstrap starting.
I20260928 02:47:59.474974 17458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee: Neither blocks nor log segments found. Creating new log.
I20260928 02:47:59.475246 17458 log.cc:824] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee: Log is configured to *not* fsync() on all Append() calls
I20260928 02:47:59.475888 17458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee: No bootstrap required, opened a new log
I20260928 02:47:59.477417 17458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } }
I20260928 02:47:59.477543 17458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:47:59.477564 17458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b9e2eceaa9452292c4fd1f0c9896ee, State: Initialized, Role: FOLLOWER
I20260928 02:47:59.477762 17458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } }
I20260928 02:47:59.477823 17458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:47:59.477843 17458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:47:59.477907 17458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:47:59.478745 17458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } }
I20260928 02:47:59.478849 17458 leader_election.cc:302] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b9e2eceaa9452292c4fd1f0c9896ee; no voters:
I20260928 02:47:59.479015 17458 leader_election.cc:288] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:47:59.479056 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:47:59.479238 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [term 1 LEADER]: Becoming Leader. State: Replica: 97b9e2eceaa9452292c4fd1f0c9896ee, State: Running, Role: LEADER
I20260928 02:47:59.479252 17458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:47:59.479334 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } }
I20260928 02:47:59.479701 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } } }
I20260928 02:47:59.479795 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:59.479707 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b9e2eceaa9452292c4fd1f0c9896ee. Latest consensus state: current_term: 1 leader_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b9e2eceaa9452292c4fd1f0c9896ee" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 44605 } } }
I20260928 02:47:59.479923 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:47:59.480053 17470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:47:59.480612 17470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:47:59.480875 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17390
I20260928 02:47:59.481000 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:47:59.481148 28086 external_mini_cluster.cc:1428] Setting key cbc50ff26f7cbb76a0c4eb96aaec2afb
I20260928 02:47:59.482146 17470 catalog_manager.cc:1347] Generated new cluster ID: 08b1fbeb660244e6a9a784ea6968a981
I20260928 02:47:59.482191 17470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:47:59.483155 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:59.481569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40587 (local address 127.27.109.190:44605)
0928 02:47:59.481704 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:59.481712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:59.481747 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:47:59.481908 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:59.481914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:59.481943 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:59.482074 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:59.482724 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:59.482728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:59.482728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:59.482732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:59.482779 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:59.482790 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:59.482862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:59.482933 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:47:59.482973 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260928 02:47:59.483898 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:47:59.487646 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:47:59.485210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53959 (local address 127.27.109.190:44605)
0928 02:47:59.485246 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:47:59.485248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:47:59.486710 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 02:47:59.486782 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:47:59.486783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:47:59.486797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:47:59.486852 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:47:59.487430 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:47:59.487431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:47:59.487431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:47:59.487436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:47:59.487448 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:47:59.487452 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:47:59.487502 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:47:59.487554 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:47:59.487568 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:47:59.497794 17470 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:47:59.498009 17470 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:47:59.499749 17470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b9e2eceaa9452292c4fd1f0c9896ee: Generated new TSK 0
I20260928 02:47:59.499905 17470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:47:59.502846 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17390
2026-09-28T02:47:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:47:59 dist-test-slave-b5p5 krb5kdc[17489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:47:59 dist-test-slave-b5p5 krb5kdc[17489](info): set up 2 sockets
Sep 28 02:47:59 dist-test-slave-b5p5 krb5kdc[17489](info): commencing operation
krb5kdc: starting...
W20260928 02:48:00.530529 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:48:00.641147 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:34885
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:00.722275 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:00.722501 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:00.722527 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:00.723609 17502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:00.723646 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:00.723664 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:00.723677 17502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:00.723690 17502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:00.725178 17502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:00.725373 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:00.725546 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:00.730540 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:00.730535 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:00.730546 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:00.730681 17502 server_base.cc:1034] running on GCE node
I20260928 02:48:00.730996 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:00.731283 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:00.732443 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790563680732417 us; error 37 us; skew 500 ppm
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563680, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:00.735219 17502 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:48:00.735718 17502 webserver.cc:460] Webserver started at http://127.27.109.190:35977/ using document root <none> and password file <none>
I20260928 02:48:00.735980 17502 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:00.736027 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:00.736168 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:00.737234 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171"
format_stamp: "Formatted at 2026-09-28 02:48:00 on dist-test-slave-b5p5"
server_key: "99769de2bb8b02da44c6c60bf973f5a1"
server_key_iv: "2564cd10a3e7ce43bdc76a9a26d0c467"
server_key_version: "encryptionkey@0"
I20260928 02:48:00.737617 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171"
format_stamp: "Formatted at 2026-09-28 02:48:00 on dist-test-slave-b5p5"
server_key: "99769de2bb8b02da44c6c60bf973f5a1"
server_key_iv: "2564cd10a3e7ce43bdc76a9a26d0c467"
server_key_version: "encryptionkey@0"
I20260928 02:48:00.739045 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:48:00.739812 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:00.740003 17502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:48:00.740097 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171"
format_stamp: "Formatted at 2026-09-28 02:48:00 on dist-test-slave-b5p5"
server_key: "99769de2bb8b02da44c6c60bf973f5a1"
server_key_iv: "2564cd10a3e7ce43bdc76a9a26d0c467"
server_key_version: "encryptionkey@0"
I20260928 02:48:00.740173 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:00.752473 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:00.754871 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:00.755038 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:00.759227 17502 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:34885
I20260928 02:48:00.759305 17569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:34885 every 8 connection(s)
I20260928 02:48:00.759667 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:00.760522 17570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:00.762915 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171: Bootstrap starting.
I20260928 02:48:00.763451 17570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:00.763693 17570 log.cc:824] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:00.764380 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171: No bootstrap required, opened a new log
I20260928 02:48:00.765578 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17502
I20260928 02:48:00.765699 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:48:00.765892 28086 external_mini_cluster.cc:1428] Setting key b35cb7c891a128f06eecec21d359df8b
I20260928 02:48:00.766353 17570 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } }
I20260928 02:48:00.766564 17570 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:00.766642 17570 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea7a8006e9d48b8b1ef7e9a64bcb171, State: Initialized, Role: FOLLOWER
I20260928 02:48:00.766789 17570 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } }
I20260928 02:48:00.766875 17570 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:00.766907 17570 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:00.766953 17570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:00.767688 17570 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } }
I20260928 02:48:00.767800 17570 leader_election.cc:302] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea7a8006e9d48b8b1ef7e9a64bcb171; no voters:
I20260928 02:48:00.767949 17570 leader_election.cc:288] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:00.768023 17575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:00.768196 17570 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:00.768206 17575 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [term 1 LEADER]: Becoming Leader. State: Replica: eea7a8006e9d48b8b1ef7e9a64bcb171, State: Running, Role: LEADER
I20260928 02:48:00.768332 17575 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } }
I20260928 02:48:00.768668 17575 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea7a8006e9d48b8b1ef7e9a64bcb171. Latest consensus state: current_term: 1 leader_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } } }
I20260928 02:48:00.768757 17575 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:00.768677 17576 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea7a8006e9d48b8b1ef7e9a64bcb171" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 34885 } } }
I20260928 02:48:00.768994 17576 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:00.769461 17584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:00.770057 17584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:00.771544 17584 catalog_manager.cc:1347] Generated new cluster ID: 215e1c3213b643a398f1fd899e9988d7
I20260928 02:48:00.771625 17584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:00.778113 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:00.766430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58977 (local address 127.27.109.190:34885)
0928 02:48:00.766657 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:00.766667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:00.766701 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:48:00.766830 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:00.766837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:00.766895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:00.767059 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:00.767695 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:00.769135 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:00.770291 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:00.770932 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:00.771032 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:00.773988 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:00.773993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:00.773994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:00.774006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:00.776023 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:00.776426 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:00.776430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:00.776430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:00.776543 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:00.776762 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:00.776763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:00.776763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:00.777077 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:00.777236 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:00.777742 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 02:48:00.777881 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":113,"threads_started":1}
I20260928 02:48:00.779261 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:00.787441 17584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:00.788054 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:00.780739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37881 (local address 127.27.109.190:34885)
0928 02:48:00.780774 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:00.780776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:00.780891 (+ 115us) server_negotiation.cc:407] Connection header received
0928 02:48:00.780984 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:00.780985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:00.781004 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:00.781034 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:00.781711 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:00.782449 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:00.783443 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:00.783755 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:00.783828 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:00.785146 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:00.785147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:00.785148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:00.785155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:00.786751 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:00.787149 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:00.787150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:00.787150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:00.787209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:00.787436 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:00.787437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:00.787437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:00.787614 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:00.787733 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:00.787833 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:48:00.787914 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:48:00.788513 17584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:00.793946 17584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea7a8006e9d48b8b1ef7e9a64bcb171: Generated new TSK 0
I20260928 02:48:00.794122 17584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:00.799127 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17502
2026-09-28T02:48:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17601](info): set up 2 sockets
Sep 28 02:48:00 dist-test-slave-b5p5 krb5kdc[17601](info): commencing operation
krb5kdc: starting...
W20260928 02:48:01.830515 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:48:01 dist-test-slave-b5p5 krb5kdc[17601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:48:01.938946 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45429
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:45429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:02.016176 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:02.016352 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:02.016373 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:02.017474 17614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:02.017513 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:02.017529 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:02.017580 17614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:02.017596 17614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:02.019011 17614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:45429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:45429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:02.019243 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:02.019446 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:02.024044 17619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:02.024134 17622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:02.024076 17620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:02.024216 17614 server_base.cc:1034] running on GCE node
I20260928 02:48:02.024494 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:02.024747 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:02.025910 17614 hybrid_clock.cc:648] HybridClock initialized: now 1790563682025901 us; error 38 us; skew 500 ppm
Sep 28 02:48:02 dist-test-slave-b5p5 krb5kdc[17601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:02.028326 17614 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:48:02.028753 17614 webserver.cc:460] Webserver started at http://127.27.109.190:42359/ using document root <none> and password file <none>
I20260928 02:48:02.028973 17614 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:02.029093 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:02.029228 17614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:02.030184 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "93fe0638f53a45c68d65fd1589b89345"
format_stamp: "Formatted at 2026-09-28 02:48:02 on dist-test-slave-b5p5"
server_key: "aaefb1363c5c159f20363d3813395b2c"
server_key_iv: "c108c45ef97121857641bcf43df7419e"
server_key_version: "encryptionkey@0"
I20260928 02:48:02.030622 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "93fe0638f53a45c68d65fd1589b89345"
format_stamp: "Formatted at 2026-09-28 02:48:02 on dist-test-slave-b5p5"
server_key: "aaefb1363c5c159f20363d3813395b2c"
server_key_iv: "c108c45ef97121857641bcf43df7419e"
server_key_version: "encryptionkey@0"
I20260928 02:48:02.032060 17614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:48:02.032675 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:02.032842 17614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:48:02.032900 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "93fe0638f53a45c68d65fd1589b89345"
format_stamp: "Formatted at 2026-09-28 02:48:02 on dist-test-slave-b5p5"
server_key: "aaefb1363c5c159f20363d3813395b2c"
server_key_iv: "c108c45ef97121857641bcf43df7419e"
server_key_version: "encryptionkey@0"
I20260928 02:48:02.032948 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:02.041427 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:02.043349 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:02.043448 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:02.047284 17614 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:45429
I20260928 02:48:02.047333 17681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:45429 every 8 connection(s)
I20260928 02:48:02.047642 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:02.048439 17682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:02.050609 17682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345: Bootstrap starting.
I20260928 02:48:02.051141 17682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:02.051414 17682 log.cc:824] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:02.052173 17682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345: No bootstrap required, opened a new log
I20260928 02:48:02.052646 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17614
I20260928 02:48:02.052772 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:48:02.052932 28086 external_mini_cluster.cc:1428] Setting key 80c59b1c16763fb50a1c171239137106
I20260928 02:48:02.054551 17682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } }
I20260928 02:48:02.054687 17682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:02.054708 17682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93fe0638f53a45c68d65fd1589b89345, State: Initialized, Role: FOLLOWER
I20260928 02:48:02.054834 17682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } }
I20260928 02:48:02.054893 17682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:02.054942 17682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:02.054987 17682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:02.055474 17682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } }
I20260928 02:48:02.055553 17682 leader_election.cc:302] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93fe0638f53a45c68d65fd1589b89345; no voters:
I20260928 02:48:02.055766 17682 leader_election.cc:288] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:02.055784 17687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:02.055898 17687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [term 1 LEADER]: Becoming Leader. State: Replica: 93fe0638f53a45c68d65fd1589b89345, State: Running, Role: LEADER
I20260928 02:48:02.056020 17682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:02.056056 17687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } }
I20260928 02:48:02.056463 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93fe0638f53a45c68d65fd1589b89345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } } }
I20260928 02:48:02.056473 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93fe0638f53a45c68d65fd1589b89345. Latest consensus state: current_term: 1 leader_uuid: "93fe0638f53a45c68d65fd1589b89345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93fe0638f53a45c68d65fd1589b89345" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 45429 } } }
I20260928 02:48:02.056638 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:02.056628 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:02.056972 17696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:02.057739 17696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:02.059288 17696 catalog_manager.cc:1347] Generated new cluster ID: 6fa78d1b896149a5b6e36e3e6dfb7990
I20260928 02:48:02.059269 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:02.053422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50251 (local address 127.27.109.190:45429)
0928 02:48:02.053608 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:02.053616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:02.053649 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:48:02.054070 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:02.054077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:02.054122 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:02.054229 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:02.054517 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:02.055955 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:02.057075 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:02.057712 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:02.057832 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:02.058532 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:02.058536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:02.058536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:48:02.058731 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:02.058769 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:48:02.058784 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:02.058876 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:02.058957 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:48:02.059017 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260928 02:48:02.059384 17696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:02.060206 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:02.069281 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:02.062621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59403 (local address 127.27.109.190:45429)
0928 02:48:02.062684 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:02.062686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:02.066658 (+ 3972us) server_negotiation.cc:407] Connection header received
0928 02:48:02.066727 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:02.066728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:02.066741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:02.066805 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:02.066960 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:02.067436 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:02.068152 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:02.068352 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:02.068427 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:02.069061 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:02.069062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:02.069062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:48:02.069067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:02.069079 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:48:02.069083 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:02.069129 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:02.069177 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:48:02.069195 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:48:02.096975 17696 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:02.097481 17696 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:02.101799 17696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93fe0638f53a45c68d65fd1589b89345: Generated new TSK 0
I20260928 02:48:02.101989 17696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:02.104291 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17614
2026-09-28T02:48:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:02 dist-test-slave-b5p5 krb5kdc[17713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:02 dist-test-slave-b5p5 krb5kdc[17713](info): set up 2 sockets
Sep 28 02:48:02 dist-test-slave-b5p5 krb5kdc[17713](info): commencing operation
krb5kdc: starting...
W20260928 02:48:03.138819 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:48:03 dist-test-slave-b5p5 krb5kdc[17713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:48:03.249593 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:42953
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:03.327718 17726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:03.327909 17726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:03.327931 17726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:03.329110 17726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:03.329157 17726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:03.329173 17726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:03.329226 17726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:03.329244 17726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:03.330681 17726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:03.330928 17726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:03.331143 17726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:03.335788 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:03.335788 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:03.335951 17726 server_base.cc:1034] running on GCE node
W20260928 02:48:03.335836 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:03.336262 17726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:03.336546 17726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:03.337708 17726 hybrid_clock.cc:648] HybridClock initialized: now 1790563683337694 us; error 48 us; skew 500 ppm
Sep 28 02:48:03 dist-test-slave-b5p5 krb5kdc[17713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563683, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:03.340119 17726 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:48:03.340575 17726 webserver.cc:460] Webserver started at http://127.27.109.190:35201/ using document root <none> and password file <none>
I20260928 02:48:03.340760 17726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:03.340801 17726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:03.340883 17726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:03.341804 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01"
format_stamp: "Formatted at 2026-09-28 02:48:03 on dist-test-slave-b5p5"
server_key: "91431844e0d02cd8c61f94c5c577cdaa"
server_key_iv: "cbd5766641ad50ec7c88ab3583388a2c"
server_key_version: "encryptionkey@0"
I20260928 02:48:03.342087 17726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01"
format_stamp: "Formatted at 2026-09-28 02:48:03 on dist-test-slave-b5p5"
server_key: "91431844e0d02cd8c61f94c5c577cdaa"
server_key_iv: "cbd5766641ad50ec7c88ab3583388a2c"
server_key_version: "encryptionkey@0"
I20260928 02:48:03.343389 17726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:48:03.344129 17741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:03.344348 17726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:48:03.344434 17726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01"
format_stamp: "Formatted at 2026-09-28 02:48:03 on dist-test-slave-b5p5"
server_key: "91431844e0d02cd8c61f94c5c577cdaa"
server_key_iv: "cbd5766641ad50ec7c88ab3583388a2c"
server_key_version: "encryptionkey@0"
I20260928 02:48:03.344487 17726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:03.347636 17726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:03.349517 17726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:03.349632 17726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:03.353307 17726 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:42953
I20260928 02:48:03.353334 17793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:42953 every 8 connection(s)
I20260928 02:48:03.353648 17726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:03.354633 17794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:03.356717 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01: Bootstrap starting.
I20260928 02:48:03.357230 17794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:03.357448 17794 log.cc:824] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:03.358075 17794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01: No bootstrap required, opened a new log
I20260928 02:48:03.359675 17794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } }
I20260928 02:48:03.359805 17794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:48:03.359827 17794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fe3ab62aa04e1d91690ec3a2b6ce01, State: Initialized, Role: FOLLOWER
I20260928 02:48:03.359953 17794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } }
I20260928 02:48:03.360037 17794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:48:03.360060 17794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:48:03.360087 17794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:03.360831 17794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } }
I20260928 02:48:03.360914 17794 leader_election.cc:302] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95fe3ab62aa04e1d91690ec3a2b6ce01; no voters:
I20260928 02:48:03.361063 17794 leader_election.cc:288] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:48:03.361079 17797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:03.361200 17797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [term 1 LEADER]: Becoming Leader. State: Replica: 95fe3ab62aa04e1d91690ec3a2b6ce01, State: Running, Role: LEADER
I20260928 02:48:03.361330 17794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:03.361335 17797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } }
I20260928 02:48:03.361683 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } } }
I20260928 02:48:03.361683 17798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95fe3ab62aa04e1d91690ec3a2b6ce01. Latest consensus state: current_term: 1 leader_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fe3ab62aa04e1d91690ec3a2b6ce01" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 42953 } } }
I20260928 02:48:03.361845 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:03.361860 17798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:03.362134 17804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:03.362703 17804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:03.363473 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17726
I20260928 02:48:03.363593 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:48:03.363790 28086 external_mini_cluster.cc:1428] Setting key bb69326ecafa06f2ec35beefef5de780
I20260928 02:48:03.364692 17804 catalog_manager.cc:1347] Generated new cluster ID: a24b7847ac454433a9a558c3e8f19e8b
I20260928 02:48:03.364749 17804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:03.368284 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:03.364361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42115 (local address 127.27.109.190:42953)
0928 02:48:03.366796 (+ 2435us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:03.366804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:03.366836 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:48:03.366945 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:03.366952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:03.366980 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:03.367118 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:03.367760 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:03.367763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:03.367764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:48:03.367768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:03.367810 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:48:03.367824 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:03.367893 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:03.367958 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:48:03.368095 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2390,"thread_start_us":69,"threads_started":1}
I20260928 02:48:03.369160 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:03.371768 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:03.370696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42923 (local address 127.27.109.190:42953)
0928 02:48:03.370734 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:03.370736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:03.370904 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:48:03.370953 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:03.370954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:03.370959 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:03.371020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:03.371584 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:03.371585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:03.371585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:48:03.371588 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:03.371599 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:48:03.371603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:03.371651 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:03.371696 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:48:03.371708 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:48:03.378072 17804 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:03.378223 17804 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:03.381661 17804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95fe3ab62aa04e1d91690ec3a2b6ce01: Generated new TSK 0
I20260928 02:48:03.381835 17804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:03.386296 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17726
2026-09-28T02:48:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1282 ms)
[----------] 18 tests from AuthTokenIssuingTest (23216 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:03 dist-test-slave-b5p5 krb5kdc[17825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:03 dist-test-slave-b5p5 krb5kdc[17825](info): set up 2 sockets
Sep 28 02:48:03 dist-test-slave-b5p5 krb5kdc[17825](info): commencing operation
krb5kdc: starting...
W20260928 02:48:04.410648 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:48:04.521013 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37637
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:04.599123 17838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:04.599363 17838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:04.599395 17838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:04.599438 17838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:04.600550 17838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:04.600615 17838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:04.600636 17838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:04.600656 17838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:04.600687 17838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:04.602103 17838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:04.602373 17838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:04.602598 17838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:04.607264 17843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.607276 17844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.607290 17846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:04.607307 17838 server_base.cc:1034] running on GCE node
I20260928 02:48:04.607642 17838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:04.607916 17838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:04.609068 17838 hybrid_clock.cc:648] HybridClock initialized: now 1790563684609056 us; error 28 us; skew 500 ppm
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:04.615130 17838 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:48:04.615536 17838 webserver.cc:460] Webserver started at http://127.27.109.190:44573/ using document root <none> and password file <none>
I20260928 02:48:04.615760 17838 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:04.615825 17838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:04.615968 17838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:04.616968 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "5ed73645ed3541c8adb308af9959c288"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "77d149f20f37e57798e1978d23b6fe77"
server_key_iv: "3326ace08bea69f7222153bbe1113433"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.617321 17838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "5ed73645ed3541c8adb308af9959c288"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "77d149f20f37e57798e1978d23b6fe77"
server_key_iv: "3326ace08bea69f7222153bbe1113433"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.618604 17838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:48:04.619304 17853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:04.619464 17838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:04.619557 17838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "5ed73645ed3541c8adb308af9959c288"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "77d149f20f37e57798e1978d23b6fe77"
server_key_iv: "3326ace08bea69f7222153bbe1113433"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.619639 17838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:04.646026 17838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:04.648265 17838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:04.648373 17838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:04.652125 17838 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:37637
I20260928 02:48:04.652161 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:37637 every 8 connection(s)
I20260928 02:48:04.652534 17838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:04.653385 17906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:04.655099 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17838
I20260928 02:48:04.655201 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:48:04.654949 17906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:04.655360 28086 external_mini_cluster.cc:1428] Setting key 5dfb63d8251dcf5db2cbbda7099cd45d
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.189@KRBTEST.COM; defaulting to no policy
I20260928 02:48:04.664793 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.655735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.27.109.190:37637)
0928 02:48:04.655931 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.655934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.655998 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:48:04.656799 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.656801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.656824 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.656929 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.657444 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.658382 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.659258 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.659568 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.659692 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.661599 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.661606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.661606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.661612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.663194 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.663624 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.663627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.663627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.663708 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.663929 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.663930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.663931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.664118 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.664210 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.664417 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.664546 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260928 02:48:04.664794 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.655672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37637 (local address 127.0.0.1:59802)
0928 02:48:04.655873 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.655919 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.656624 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.656959 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.656962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.657200 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.657347 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.657350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.658496 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.658497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.659146 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.659148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.659665 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.659666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.659726 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.660219 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.660227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.661502 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.663292 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.663298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.663300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.663530 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.663777 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.663777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.663778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.663864 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.664190 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.664214 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.664272 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.664386 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.664605 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":94,"thread_start_us":86,"threads_started":1}
I20260928 02:48:04.665571 17906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.189@KRBTEST.COM" created.
I20260928 02:48:04.665930 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.665799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39700)
0928 02:48:04.665850 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.665896 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:48:04.666069 17854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
I20260928 02:48:04.666512 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.666368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39710)
0928 02:48:04.666432 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.666479 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:48:04.666625 17906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:44453: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
I20260928 02:48:04.680538 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:44453
--webserver_interface=127.27.109.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:04.715852 17906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } attempt: 1
I20260928 02:48:04.716332 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.716196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39712)
0928 02:48:04.716259 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.716301 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:48:04.716691 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.716588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39716)
0928 02:48:04.716635 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.716661 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:48:04.716777 17906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:44453: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
W20260928 02:48:04.759893 17913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:04.760138 17913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:04.760167 17913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:04.760218 17913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:04.761397 17913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:04.761469 17913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:04.761492 17913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:04.761512 17913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:04.761543 17913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:04.762991 17913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:04.763267 17913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:04.763487 17913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:04.768273 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.768285 17919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.768292 17918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:04.768384 17913 server_base.cc:1034] running on GCE node
I20260928 02:48:04.768764 17913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:04.769085 17913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:04.770251 17913 hybrid_clock.cc:648] HybridClock initialized: now 1790563684770237 us; error 25 us; skew 500 ppm
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:04.772735 17913 init.cc:377] Logged in from keytab as kudu/127.27.109.189@KRBTEST.COM (short username kudu)
I20260928 02:48:04.773097 17913 webserver.cc:460] Webserver started at http://127.27.109.189:41653/ using document root <none> and password file <none>
I20260928 02:48:04.773288 17913 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:04.773327 17913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:04.773412 17913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:04.774441 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data/instance:
uuid: "de2676d4baec4ec6b626c4237ae7ec83"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "66fd51d27e8d184225345bf1dd96479a"
server_key_iv: "d032bdd0e3ec844186f0a7e0b350db27"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.774816 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal/instance:
uuid: "de2676d4baec4ec6b626c4237ae7ec83"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "66fd51d27e8d184225345bf1dd96479a"
server_key_iv: "d032bdd0e3ec844186f0a7e0b350db27"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.776042 17913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:48:04.776746 17928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:04.776942 17913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:48:04.777017 17913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "de2676d4baec4ec6b626c4237ae7ec83"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "66fd51d27e8d184225345bf1dd96479a"
server_key_iv: "d032bdd0e3ec844186f0a7e0b350db27"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.777092 17913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:04.784911 17906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } attempt: 2
I20260928 02:48:04.785298 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.785148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39718)
0928 02:48:04.785213 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.785262 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:48:04.785636 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.785555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39722)
0928 02:48:04.785597 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.785620 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:48:04.785748 17906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:44453: Network error: Client connection negotiation failed: client connection to 127.27.109.189:44453: connect: Connection refused (error 111)
I20260928 02:48:04.796132 17913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:04.798337 17913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:04.798519 17913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:04.802124 17913 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.189:44453
I20260928 02:48:04.802192 17980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.189:44453 every 8 connection(s)
I20260928 02:48:04.802563 17913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
I20260928 02:48:04.803408 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:04.804780 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17913
I20260928 02:48:04.804898 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-1/wal/instance
I20260928 02:48:04.805076 28086 external_mini_cluster.cc:1428] Setting key 4cd77bf854a732680f1e71dbf7bc6db0
I20260928 02:48:04.805018 17981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.188@KRBTEST.COM; defaulting to no policy
I20260928 02:48:04.814355 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.805758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.27.109.190:37637)
0928 02:48:04.805821 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.805823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.805990 (+ 167us) server_negotiation.cc:407] Connection header received
0928 02:48:04.806496 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.806497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.806510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.806569 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.807173 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.807907 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.809065 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.809428 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.809511 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.811565 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.811566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.811567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.811574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.813148 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.813525 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.813526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.813526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.813576 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.813820 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.813821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.813821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.813937 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.813984 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.814234 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.814293 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:48:04.814559 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.805719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37637 (local address 127.0.0.1:59812)
0928 02:48:04.805867 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.805906 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.806358 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.806639 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.806643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.806886 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.807067 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.807071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.808017 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.808018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.808958 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.808960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.809519 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.809520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.809624 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.810113 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.810121 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.811473 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.813234 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.813240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.813242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.813454 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.813636 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.813637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.813637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.813730 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.813996 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.814002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.814113 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.814200 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.814310 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260928 02:48:04.815193 17981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.188@KRBTEST.COM" created.
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:04.822676 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.815467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39738)
0928 02:48:04.815582 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.815594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.815639 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.815869 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.815870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.815975 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.816016 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.816018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.817172 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.817173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.817662 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.817664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.818154 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.818155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.818223 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.818643 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.818648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.819946 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.821561 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.821562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.821562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.821725 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.821961 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.821961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.821962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.822021 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.822331 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.822332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.822368 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.822439 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.822555 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 02:48:04.822747 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.815533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.27.109.189:44453)
0928 02:48:04.815696 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.815702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.815738 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:48:04.815760 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.815762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.815775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.815851 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.816112 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.817083 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.817760 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.818065 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.818154 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.820040 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.820047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.820048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.820054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.821463 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.821815 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.821818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.821818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.821879 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.822119 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.822120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.822120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.822261 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.822352 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.822485 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.822606 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 02:48:04.823518 17981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:04.823823 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.823742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40360)
0928 02:48:04.823769 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.823807 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 02:48:04.823948 17931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
I20260928 02:48:04.824368 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.824263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40366)
0928 02:48:04.824314 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.824340 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:48:04.824481 17981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:43985: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
I20260928 02:48:04.829139 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:43985
--webserver_interface=127.27.109.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:04.873680 17981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } attempt: 1
I20260928 02:48:04.874146 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.874021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40374)
0928 02:48:04.874082 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.874120 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:48:04.874540 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.874415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40386)
0928 02:48:04.874456 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.874521 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:48:04.874652 17981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:43985: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
I20260928 02:48:04.876922 17906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } attempt: 3
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:04.884303 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.877202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39746)
0928 02:48:04.877265 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.877276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.877332 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.877477 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.877477 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.877615 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.877674 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.877676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.878614 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.878616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.879124 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.879125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.879625 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.879626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.879689 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.880109 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.880113 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.881511 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.883342 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.883343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.883343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.883504 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.883767 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.883768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.883768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.883824 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.884082 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.884083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.884120 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.884172 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.884241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:48:04.884356 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.877279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.27.109.189:44453)
0928 02:48:04.877321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.877323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.877336 (+ 13us) server_negotiation.cc:407] Connection header received
0928 02:48:04.877396 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.877397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.877409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.877463 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.877744 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.878533 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.879203 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.879538 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.879617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.881599 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.881601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.881602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.881609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.883251 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.883619 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.883621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.883621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.883690 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.883888 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.883889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.883889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.884018 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.884087 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.884194 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.884272 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:48:04.884757 17906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:04.885093 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.884994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40396)
0928 02:48:04.885040 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.885067 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:48:04.885473 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.885386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40398)
0928 02:48:04.885428 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.885452 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:48:04.885591 17906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:43985: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
W20260928 02:48:04.911134 17988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:04.911317 17988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:04.911334 17988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:04.911353 17988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:04.912447 17988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:04.912482 17988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:04.912495 17988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:04.912508 17988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:04.912520 17988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:04.913883 17988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:37637,127.27.109.189:44453,127.27.109.188:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:04.914055 17988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:04.914299 17988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:48:04.916736 17906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } attempt: 1
I20260928 02:48:04.917160 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.917025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40414)
0928 02:48:04.917090 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.917130 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:48:04.917515 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.917423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40426)
0928 02:48:04.917469 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.917495 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:48:04.917622 17906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:43985: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
W20260928 02:48:04.919165 17993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.919170 17994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:04.919191 17996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:04.919389 17988 server_base.cc:1034] running on GCE node
I20260928 02:48:04.919600 17988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:04.919870 17988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:04.921036 17988 hybrid_clock.cc:648] HybridClock initialized: now 1790563684921030 us; error 33 us; skew 500 ppm
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:04.923549 17988 init.cc:377] Logged in from keytab as kudu/127.27.109.188@KRBTEST.COM (short username kudu)
I20260928 02:48:04.923947 17988 webserver.cc:460] Webserver started at http://127.27.109.188:38095/ using document root <none> and password file <none>
I20260928 02:48:04.924170 17988 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:04.924247 17988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:04.924404 17988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:04.925372 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data/instance:
uuid: "2e78b766ba924367bf436a89ee79483e"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "ffc61151f6f1d0d58bb36e9a55459c6a"
server_key_iv: "db7f02b512645991166a61a0c3c4ebc3"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.925709 17988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal/instance:
uuid: "2e78b766ba924367bf436a89ee79483e"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "ffc61151f6f1d0d58bb36e9a55459c6a"
server_key_iv: "db7f02b512645991166a61a0c3c4ebc3"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.927057 17988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:48:04.927771 18003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:04.927939 17988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:48:04.928035 17988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "2e78b766ba924367bf436a89ee79483e"
format_stamp: "Formatted at 2026-09-28 02:48:04 on dist-test-slave-b5p5"
server_key: "ffc61151f6f1d0d58bb36e9a55459c6a"
server_key_iv: "db7f02b512645991166a61a0c3c4ebc3"
server_key_version: "encryptionkey@0"
I20260928 02:48:04.928112 17988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:04.942826 17981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } attempt: 2
I20260928 02:48:04.943252 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.943127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40432)
0928 02:48:04.943191 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.943227 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:48:04.943585 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.943493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40440)
0928 02:48:04.943537 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.943561 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:48:04.943693 17981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:43985: Network error: Client connection negotiation failed: client connection to 127.27.109.188:43985: connect: Connection refused (error 111)
I20260928 02:48:04.957538 17988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:04.959802 17988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:04.959960 17988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:04.963749 17988 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.188:43985
I20260928 02:48:04.963783 18055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.188:43985 every 8 connection(s)
I20260928 02:48:04.964110 17988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
I20260928 02:48:04.964921 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:04.966387 18056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:04.973448 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 17988
I20260928 02:48:04.973582 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563399140758-28086-0/minicluster-data/master-2/wal/instance
I20260928 02:48:04.973776 28086 external_mini_cluster.cc:1428] Setting key d5ec3b7bdcdbfaffa19944b07f6fb640
I20260928 02:48:04.974622 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.967024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.27.109.190:37637)
0928 02:48:04.967089 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.967091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.967235 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:48:04.967656 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.967657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.967665 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.967721 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.968277 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.968742 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.969744 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.969926 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.969994 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.971934 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.971936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.971936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.971942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.973082 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.973484 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.973486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.973486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.973568 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.973837 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.973838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.973839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.973976 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.974051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.974407 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.974515 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:48:04.974803 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.966966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:37637 (local address 127.0.0.1:59822)
0928 02:48:04.967109 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.967148 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.967578 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.967769 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.967772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.967994 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.968170 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.968173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.968825 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.968826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.969652 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.969654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.970008 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.970009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.970103 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.970590 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.970598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.971854 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.973151 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.973156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.973158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.973381 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.973629 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.973629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.973630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.973770 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.974054 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.974081 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.974230 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.974373 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.974560 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260928 02:48:04.975330 18056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:48:04.982573 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.975665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.27.109.189:44453)
0928 02:48:04.975716 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.975718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.975756 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:48:04.975823 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.975824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.975831 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.975872 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.976121 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.976675 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.977569 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.977889 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.977954 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.979885 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.979886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.979887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.979896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.981389 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.981831 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.981832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.981832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.981883 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.982081 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.982081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.982082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.982180 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.982230 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.982408 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.982505 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:48:04.982623 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.975581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:44453 (local address 127.0.0.1:39762)
0928 02:48:04.975711 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.975721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.975769 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.975876 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.975876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.976015 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.976060 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.976061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.976753 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.976754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.977483 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.977486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.977974 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.977975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.978045 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.978421 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.978426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.979804 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.981468 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.981469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.981470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.981754 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.981950 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.981950 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.981951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.982018 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.982245 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.982246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.982289 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.982392 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.982509 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 02:48:04.982950 18056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:04.983834 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.974174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.27.109.188:43985)
0928 02:48:04.974320 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.974324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.974393 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:48:04.974449 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.974452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.974489 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.974570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.975296 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.976282 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.977306 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.977749 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.977831 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.980518 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.980525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.980526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.980532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.982519 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.982913 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.982916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.982916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.982973 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.983206 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.983207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.983207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.983379 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.983570 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.983615 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.983708 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:04.991300 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.983216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40464)
0928 02:48:04.983249 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.983259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.983296 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.983550 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.983550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.983708 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.983736 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.983737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.984590 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.984591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.985163 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.985166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.985499 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.985500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.985558 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.986464 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.986469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.987912 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.990327 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.990328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.990329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.990582 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.990771 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.990771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.990772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.990836 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.991036 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.991037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.991084 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:48:04.991150 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:48:04.991244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 02:48:04.991515 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.983259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.27.109.188:43985)
0928 02:48:04.983411 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.983429 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.983436 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:48:04.983453 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.983454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.983462 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.983528 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.983813 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.984487 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.985242 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.985432 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.985481 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.987994 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.987996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.987996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.988003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.990239 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.990654 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.990655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.990655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.990704 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.990898 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.990898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.990898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.990979 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.991023 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.991185 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.991305 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260928 02:48:04.992331 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.984607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.27.109.190:37637)
0928 02:48:04.984650 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.984652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.984659 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:48:04.984701 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.984702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.984711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.984767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.985154 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.985750 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.986424 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.986736 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.986809 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.989314 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.989315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.989316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.989322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.991096 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.991465 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.991466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.991466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.991519 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.991852 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.991853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.991853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.991979 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.992147 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:04.992187 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:48:04.992252 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:48:04.992741 17906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } attempt: 2
I20260928 02:48:04.993052 18056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: Bootstrap starting.
I20260928 02:48:04.993803 18056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:04.994130 18056 log.cc:824] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:04.994967 18056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: No bootstrap required, opened a new log
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:48:04.997104 18056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:04.997268 18056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e78b766ba924367bf436a89ee79483e, State: Initialized, Role: FOLLOWER
I20260928 02:48:04.997392 18056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:04.997673 18063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:04.997740 18056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:04.997829 18063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 02:48:04 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:05.000198 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.992967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40472)
0928 02:48:04.993055 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:04.993064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:04.993097 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:04.993354 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:04.993355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:04.993511 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:04.993541 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.993543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.994311 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.994312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.994989 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:04.994991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.995374 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.995375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:04.995429 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.995835 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:04.995840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:04.997243 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:04.999111 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.999112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.999112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.999328 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.999501 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:04.999502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:04.999503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:04.999562 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:04.999897 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:04.999898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:04.999943 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:48:05.000016 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:48:05.000105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:48:05.000216 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.993051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.27.109.188:43985)
0928 02:48:04.993238 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.993240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.993247 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:48:04.993264 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.993264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.993272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.993595 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.993627 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.994212 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.995096 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.995302 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.995376 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.997363 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.997364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.997365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.997372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:04.999004 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.999388 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.999389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.999389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.999446 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:04.999642 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:04.999643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:04.999644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:04.999801 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:04.999866 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.000058 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.000137 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
W20260928 02:48:05.001021 18074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:05.001118 18074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:05.002197 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:04.993175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.27.109.189:44453)
0928 02:48:04.993230 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:04.993232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:04.993240 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:48:04.993263 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:04.993264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:04.993273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:04.993326 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:04.993973 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.994674 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.995535 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:04.995820 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:04.995882 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:04.999081 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:04.999083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:04.999083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:04.999091 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:05.001010 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.001405 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.001406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.001407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.001469 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.001696 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.001697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.001697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.001822 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:05.002000 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.002038 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.002109 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:48:05.001240 17988 master_runner.cc:186] Error getting master registration for 127.27.109.190:37637: OK, instance_id { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" instance_seqno: 1790563684651060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:48:05.003407 17988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:05.003611 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:05.004895 17906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: Bootstrap starting.
I20260928 02:48:05.004643 17880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5ed73645ed3541c8adb308af9959c288"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59824
I20260928 02:48:05.005586 17906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:05.005879 17906 log.cc:824] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:05.006649 17906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: No bootstrap required, opened a new log
I20260928 02:48:05.007623 17906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:05.007732 17906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed73645ed3541c8adb308af9959c288, State: Initialized, Role: FOLLOWER
I20260928 02:48:05.007831 17906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:05.008140 17906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:05.008121 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.008229 17880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:48:05.008244 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.008329 17880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:05.009163 17880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:05.009392 17880 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [CANDIDATE]: Term 1 election: Requested vote from peers de2676d4baec4ec6b626c4237ae7ec83 (127.27.109.189:44453), 2e78b766ba924367bf436a89ee79483e (127.27.109.188:43985)
I20260928 02:48:05.009630 18030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ed73645ed3541c8adb308af9959c288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2e78b766ba924367bf436a89ee79483e"
I20260928 02:48:05.009712 18030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:05.010382 18030 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed73645ed3541c8adb308af9959c288 in term 1.
I20260928 02:48:05.010624 17856 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e78b766ba924367bf436a89ee79483e, 5ed73645ed3541c8adb308af9959c288; no voters:
I20260928 02:48:05.010841 18077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:05.010952 18077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed73645ed3541c8adb308af9959c288, State: Running, Role: LEADER
I20260928 02:48:05.011036 18077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
W20260928 02:48:05.011250 18088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:05.011284 18088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:05.011436 18077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed73645ed3541c8adb308af9959c288. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.011499 18077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:05.011936 17838 master_runner.cc:186] Error getting master registration for 127.27.109.189:44453: OK, instance_id { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" instance_seqno: 1790563684801145 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:48:05.014149 17838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:05.015130 18096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:05.015334 18096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:05.021668 18030 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [term 1 FOLLOWER]: Refusing update from remote peer 5ed73645ed3541c8adb308af9959c288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:05.022032 18090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:05.023530 18063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed73645ed3541c8adb308af9959c288. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.023615 18063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.025352 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:05.014966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.27.109.188:43985)
0928 02:48:05.015000 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.015002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.016166 (+ 1164us) server_negotiation.cc:407] Connection header received
0928 02:48:05.016269 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.016269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.016281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.016316 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.017077 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.017820 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.018924 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.019221 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.019265 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:05.021013 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:05.021015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:05.021015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:05.021022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:05.022919 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.023843 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.023845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.023845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.023913 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.024474 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.024475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.024475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.024612 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:05.024680 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.025181 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.025259 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:48:05.031674 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:05.014592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.27.109.190:37637)
0928 02:48:05.014639 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.014641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.015578 (+ 937us) server_negotiation.cc:407] Connection header received
0928 02:48:05.015802 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.015803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.015825 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.015854 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.017136 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.017858 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.019395 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.019696 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.019735 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:05.026586 (+ 6851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:05.026588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:05.026588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:05.026594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:05.028409 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.030034 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.030035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.030036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.030097 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.031009 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.031011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.031011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.031155 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:05.031252 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.031485 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.031566 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:48:05.032774 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.032878 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:05.033365 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:05.015487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.27.109.189:44453)
0928 02:48:05.015522 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.015524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.023829 (+ 8305us) server_negotiation.cc:407] Connection header received
0928 02:48:05.023962 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.023963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.023984 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.024014 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.024947 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.025686 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.027493 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.027811 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.027852 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:05.029496 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:05.029497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:05.029498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:05.029504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:05.031151 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.031818 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.031819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.031820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.031889 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.032625 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.032626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.032626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.032763 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:05.032830 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.033157 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.033264 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:48:05.033380 18063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.033452 18063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.033816 18099 mvcc.cc:205] Tried to move back new op lower bound from 7334148853845925888 to 7334148853805924352. Current Snapshot: MvccSnapshot[applied={T|T < 7334148853845925888}]
I20260928 02:48:05.034010 18101 mvcc.cc:205] Tried to move back new op lower bound from 7334148853845925888 to 7334148853805924352. Current Snapshot: MvccSnapshot[applied={T|T < 7334148853845925888}]
I20260928 02:48:05.034818 17981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } attempt: 3
I20260928 02:48:05.035063 18096 catalog_manager.cc:1347] Generated new cluster ID: c2d63c3dbba543f7892bebfb7459e456
I20260928 02:48:05.035135 18096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:48:05 dist-test-slave-b5p5 krb5kdc[17825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563684, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:48:05.043138 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:05.035050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:43985 (local address 127.0.0.1:40504)
0928 02:48:05.035125 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:05.035134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:05.035167 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:05.035368 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:05.035369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:05.035518 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:05.035551 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:05.035553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.036488 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.036488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:05.037295 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:05.037297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.037844 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.037845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:05.037915 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:05.038505 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:05.038510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:05.040082 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:05.041999 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:05.042000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:05.042001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:05.042213 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:05.042437 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:05.042438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:05.042438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:05.042524 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:05.042840 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:05.042841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:05.042895 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:48:05.042956 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:48:05.043044 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:48:05.043514 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:05.035228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.27.109.188:43985)
0928 02:48:05.035260 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.035261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.035267 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:48:05.035285 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.035285 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.035293 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.035335 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.035645 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.036383 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.037403 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:05.037727 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:05.037782 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:05.040284 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:05.040286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:05.040286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:05.040292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:05.041906 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.042283 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.042284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.042284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.042341 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:05.042605 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:05.042605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:05.042606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:05.042743 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:05.042799 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:05.043361 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 02:48:05.043431 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:48:05.044889 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: Bootstrap starting.
I20260928 02:48:05.045568 17981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:05.045866 17981 log.cc:824] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:05.050227 18096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:05.050971 18096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:05.051080 17981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: No bootstrap required, opened a new log
I20260928 02:48:05.053014 17981 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:05.053134 17981 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2676d4baec4ec6b626c4237ae7ec83, State: Initialized, Role: FOLLOWER
I20260928 02:48:05.053205 17981 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } }
I20260928 02:48:05.053489 17981 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:05.054615 17954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:05.055442 17954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [term 1 FOLLOWER]: Refusing update from remote peer 5ed73645ed3541c8adb308af9959c288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:05.055665 17955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5ed73645ed3541c8adb308af9959c288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de2676d4baec4ec6b626c4237ae7ec83"
I20260928 02:48:05.055701 18077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:05.056375 17955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed73645ed3541c8adb308af9959c288 in term 1.
W20260928 02:48:05.058058 18121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:05.058154 18121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:05.058450 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.058585 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.058670 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed73645ed3541c8adb308af9959c288. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.058866 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.058936 18106 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.059034 18106 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:05.061507 18096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288: Generated new TSK 0
I20260928 02:48:05.061650 18096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:48:05.065447 18115 mvcc.cc:205] Tried to move back new op lower bound from 7334148853960351744 to 7334148853805924352. Current Snapshot: MvccSnapshot[applied={T|T < 7334148853845925888}]
I20260928 02:48:05.066147 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ed73645ed3541c8adb308af9959c288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed73645ed3541c8adb308af9959c288" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 37637 } } peers { permanent_uuid: "de2676d4baec4ec6b626c4237ae7ec83" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 44453 } } peers { permanent_uuid: "2e78b766ba924367bf436a89ee79483e" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 43985 } } }
I20260928 02:48:05.066226 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed73645ed3541c8adb308af9959c288 [sys.catalog]: This master's current role is: LEADER
W20260928 02:48:05.107394 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.102760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.27.109.189:44453)
0928 02:48:05.102803 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.102805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.106713 (+ 3908us) server_negotiation.cc:407] Connection header received
0928 02:48:05.106834 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.106835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.106846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.106876 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.107328 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39794: BlockingRecv error: recv got EOF from 127.0.0.1:39794 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:48:05.108439 17909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.103022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.27.109.190:37637)
0928 02:48:05.103062 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.103064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.107646 (+ 4582us) server_negotiation.cc:407] Connection header received
0928 02:48:05.107940 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.107941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.107952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.107980 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.108381 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59848: BlockingRecv error: recv got EOF from 127.0.0.1:59848 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:48:05.108914 18061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.103146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.27.109.188:43985)
0928 02:48:05.103184 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.103186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.107743 (+ 4557us) server_negotiation.cc:407] Connection header received
0928 02:48:05.108043 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.108045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.108055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.108085 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.108858 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40506: BlockingRecv error: recv got EOF from 127.0.0.1:40506 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 02:48:05.111327 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.110511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.27.109.189:44453)
0928 02:48:05.110547 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.110549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.110798 (+ 249us) server_negotiation.cc:407] Connection header received
0928 02:48:05.110899 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.110900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.110916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.110945 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.111278 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39798: BlockingRecv error: recv got EOF from 127.0.0.1:39798 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:48:05.114274 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.113462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.27.109.189:44453)
0928 02:48:05.113493 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.113495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.113797 (+ 302us) server_negotiation.cc:407] Connection header received
0928 02:48:05.113894 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.113895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.113905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.113933 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.114236 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39812: BlockingRecv error: recv got EOF from 127.0.0.1:39812 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:05.118137 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.117351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.27.109.189:44453)
0928 02:48:05.117383 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.117385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.117673 (+ 288us) server_negotiation.cc:407] Connection header received
0928 02:48:05.117767 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.117768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.117777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.117805 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.118099 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39828: BlockingRecv error: recv got EOF from 127.0.0.1:39828 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:05.124063 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.123333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39836 (local address 127.27.109.189:44453)
0928 02:48:05.123365 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.123366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.123569 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:48:05.123680 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.123681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.123690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.123718 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.124025 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39836: BlockingRecv error: recv got EOF from 127.0.0.1:39836 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:05.133913 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.133140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.27.109.189:44453)
0928 02:48:05.133173 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.133175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.133378 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:48:05.133475 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.133475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.133485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.133513 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.133861 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39852: BlockingRecv error: recv got EOF from 127.0.0.1:39852 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:05.152069 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.151169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.27.109.189:44453)
0928 02:48:05.151210 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.151212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.151460 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:48:05.151571 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.151573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.151584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.151636 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.152018 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39858: BlockingRecv error: recv got EOF from 127.0.0.1:39858 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 02:48:05.186175 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.185360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.27.109.189:44453)
0928 02:48:05.185405 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.185408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.185557 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:48:05.185653 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.185654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.185666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.185738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.186068 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39860: BlockingRecv error: recv got EOF from 127.0.0.1:39860 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:48:05.252198 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.251426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.27.109.189:44453)
0928 02:48:05.251516 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.251518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.251613 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:48:05.251700 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.251700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.251711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.251757 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.252150 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39866: BlockingRecv error: recv got EOF from 127.0.0.1:39866 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:48:05.382068 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.381428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.27.109.189:44453)
0928 02:48:05.381513 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.381515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.381569 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:48:05.381628 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.381629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.381645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.381691 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.382014 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39882: BlockingRecv error: recv got EOF from 127.0.0.1:39882 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 02:48:05.639923 17985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:05.639242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.27.109.189:44453)
0928 02:48:05.639316 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:05.639319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:05.639345 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:48:05.639423 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:05.639424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:05.639435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:05.639488 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:05.639833 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39888: BlockingRecv error: recv got EOF from 127.0.0.1:39888 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:48:06.001602 18074 catalog_manager.cc:1259] Loaded cluster ID: c2d63c3dbba543f7892bebfb7459e456
I20260928 02:48:06.001674 18074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: loading cluster ID for follower catalog manager: success
I20260928 02:48:06.002581 18074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: acquiring CA information for follower catalog manager: success
I20260928 02:48:06.002895 18074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e78b766ba924367bf436a89ee79483e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:06.058620 18121 catalog_manager.cc:1259] Loaded cluster ID: c2d63c3dbba543f7892bebfb7459e456
I20260928 02:48:06.058696 18121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: loading cluster ID for follower catalog manager: success
I20260928 02:48:06.059495 18121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: acquiring CA information for follower catalog manager: success
I20260928 02:48:06.059804 18121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de2676d4baec4ec6b626c4237ae7ec83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:06.156075 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:06.153095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.27.109.189:44453)
0928 02:48:06.153225 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:06.153228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:06.153321 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:48:06.153425 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:06.153426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:06.153502 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:06.153592 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:06.153760 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.154364 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.155235 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.155455 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.155540 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:06.155725 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:06.155844 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:06.155935 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:06.155958 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:48:06.155980 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 02:48:06.160492 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:06.157905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.27.109.188:43985)
0928 02:48:06.158022 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:06.158024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:06.158033 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:48:06.158085 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:06.158086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:06.158132 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:06.158226 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:06.158332 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.158949 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.159692 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.159907 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.159979 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:06.160145 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:06.160256 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:06.160342 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:06.160383 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:48:06.160404 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260928 02:48:06.164512 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:06.162218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.27.109.188:43985)
0928 02:48:06.162255 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:06.162257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:06.162432 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:48:06.162454 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:06.162455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:06.162487 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:06.162538 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:06.162648 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.163123 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.163868 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.164047 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.164131 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:06.164280 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:06.164309 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:06.164361 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:06.164425 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:48:06.164446 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:48:06.165026 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:06.162062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.27.109.189:44453)
0928 02:48:06.162122 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:06.162124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:06.162136 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:48:06.162224 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:06.162225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:06.162236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:06.162296 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:06.162406 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.163139 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.164232 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.164528 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.164602 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:06.164757 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:06.164788 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:06.164818 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:06.164925 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:48:06.164948 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:48:06.165347 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:06.162028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.27.109.190:37637)
0928 02:48:06.162139 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:06.162141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:06.162354 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:48:06.162404 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:06.162406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:06.162458 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:06.162556 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:06.162712 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.163440 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.164425 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:06.164724 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:06.164790 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:06.164969 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:06.165087 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:06.165163 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:06.165243 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:48:06.165270 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260928 02:48:06.167099 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17838
I20260928 02:48:06.172464 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17913
I20260928 02:48:06.177127 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 17988
2026-09-28T02:48:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2792 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:48:06 dist-test-slave-b5p5 krb5kdc[18207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:48:06 dist-test-slave-b5p5 krb5kdc[18207](info): set up 2 sockets
Sep 28 02:48:06 dist-test-slave-b5p5 krb5kdc[18207](info): commencing operation
krb5kdc: starting...
W20260928 02:48:07.198704 28086 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/test-user.keytab.
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:48:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:48:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
Entry for principal kudu/127.27.109.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab.
I20260928 02:48:07.308626 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40201
--webserver_interface=127.27.109.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:48:07.387053 18220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:07.387253 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:07.387270 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:07.387290 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:07.388343 18220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:07.388373 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:07.388387 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:07.388399 18220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:07.388411 18220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:07.389796 18220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.190.keytab
--principal=kudu/127.27.109.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.190:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:07.389972 18220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:07.390133 18220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:07.394635 18225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:07.394762 18220 server_base.cc:1034] running on GCE node
W20260928 02:48:07.394662 18226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:07.394662 18228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:07.395082 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:07.395359 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:07.396519 18220 hybrid_clock.cc:648] HybridClock initialized: now 1790563687396505 us; error 27 us; skew 500 ppm
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:07.402763 18220 init.cc:377] Logged in from keytab as kudu/127.27.109.190@KRBTEST.COM (short username kudu)
I20260928 02:48:07.403231 18220 webserver.cc:460] Webserver started at http://127.27.109.190:46143/ using document root <none> and password file <none>
I20260928 02:48:07.403463 18220 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:07.403530 18220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:07.403684 18220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:07.404702 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data/instance:
uuid: "3785239362cf405cb1c6b25e3931dc6b"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "6f6d1d8d256c856d63afbd86876a97da"
server_key_iv: "78ff4756883adbbc733a19f066f63fe7"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.405063 18220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance:
uuid: "3785239362cf405cb1c6b25e3931dc6b"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "6f6d1d8d256c856d63afbd86876a97da"
server_key_iv: "78ff4756883adbbc733a19f066f63fe7"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.406610 18220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:48:07.407573 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:07.407785 18220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:48:07.407882 18220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal
uuid: "3785239362cf405cb1c6b25e3931dc6b"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "6f6d1d8d256c856d63afbd86876a97da"
server_key_iv: "78ff4756883adbbc733a19f066f63fe7"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.407959 18220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:07.419994 18220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:07.421604 18220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:07.421708 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:07.425684 18220 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.190:40201
I20260928 02:48:07.425712 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.190:40201 every 8 connection(s)
I20260928 02:48:07.426030 18220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/data/info.pb
I20260928 02:48:07.426879 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:07.428257 18288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:07.432811 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 18220
I20260928 02:48:07.432888 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-0/wal/instance
I20260928 02:48:07.433091 28086 external_mini_cluster.cc:1428] Setting key 454737a70f46af47498597acad40bdf0
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:07.438382 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.428799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40201 (local address 127.0.0.1:32988)
0928 02:48:07.428995 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.429035 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.429520 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.429801 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.429803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.430061 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.430207 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.430211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.431277 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.431278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.431915 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.431916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.432414 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.432415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.432478 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.433044 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.433051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.434722 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.436891 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.436900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.436902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.437107 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.437408 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.437409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.437410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.437499 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.437967 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.437973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.438029 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.438105 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.438227 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260928 02:48:07.438509 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.428854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.27.109.190:40201)
0928 02:48:07.429003 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.429012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.429151 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:48:07.429675 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.429677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.429696 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.429797 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.430302 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.431166 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.432009 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.432319 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.432442 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.434809 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.434817 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.434818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.434825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.436797 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.437189 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.437192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.437192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.437298 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.437577 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.437578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.437578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.437881 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.437973 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.438177 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.438312 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260928 02:48:07.439136 18288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:07.439477 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.439375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59650)
0928 02:48:07.439420 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.439461 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:48:07.439630 18238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
I20260928 02:48:07.439985 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.439925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59654)
0928 02:48:07.439948 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.439970 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 02:48:07.440099 18288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:42113: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.109.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
Entry for principal kudu/127.27.109.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab.
I20260928 02:48:07.456725 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:42113
--webserver_interface=127.27.109.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:07.489337 18288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } attempt: 1
I20260928 02:48:07.489836 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.489696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59666)
0928 02:48:07.489746 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.489800 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 02:48:07.490275 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.490180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59672)
0928 02:48:07.490220 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.490255 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:48:07.490363 18288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:42113: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
W20260928 02:48:07.555006 18295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:07.555208 18295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:07.555225 18295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:07.555244 18295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:07.556383 18295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:07.556422 18295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:07.556437 18295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:07.556448 18295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:07.556460 18295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:07.557912 18295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.189.keytab
--principal=kudu/127.27.109.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.189:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:07.558151 18295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:07.558362 18295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:48:07.558481 18288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } attempt: 2
I20260928 02:48:07.558934 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.558767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59678)
0928 02:48:07.558862 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.558905 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:48:07.559310 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.559245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59688)
0928 02:48:07.559270 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.559294 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 02:48:07.559422 18288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.189:42113: Network error: Client connection negotiation failed: client connection to 127.27.109.189:42113: connect: Connection refused (error 111)
W20260928 02:48:07.563303 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:07.563299 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:07.563355 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:07.563539 18295 server_base.cc:1034] running on GCE node
I20260928 02:48:07.563745 18295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:07.563967 18295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:07.565131 18295 hybrid_clock.cc:648] HybridClock initialized: now 1790563687565075 us; error 50 us; skew 500 ppm
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:07.567520 18295 init.cc:377] Logged in from keytab as kudu/127.27.109.189@KRBTEST.COM (short username kudu)
I20260928 02:48:07.567902 18295 webserver.cc:460] Webserver started at http://127.27.109.189:36971/ using document root <none> and password file <none>
I20260928 02:48:07.568127 18295 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:07.568173 18295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:07.568305 18295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:07.569295 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data/instance:
uuid: "7e1cd932465f4cebb1a1d3e92edef78a"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "1bb98066d52e8c82bc4b7d6f29fbd82f"
server_key_iv: "da34472ac1ccd04518d3f8e5c5e97795"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.569628 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal/instance:
uuid: "7e1cd932465f4cebb1a1d3e92edef78a"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "1bb98066d52e8c82bc4b7d6f29fbd82f"
server_key_iv: "da34472ac1ccd04518d3f8e5c5e97795"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.570961 18295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:48:07.571632 18310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:07.571790 18295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:48:07.571882 18295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal
uuid: "7e1cd932465f4cebb1a1d3e92edef78a"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "1bb98066d52e8c82bc4b7d6f29fbd82f"
server_key_iv: "da34472ac1ccd04518d3f8e5c5e97795"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.571957 18295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:07.591126 18295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:07.592857 18295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:07.593010 18295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:07.596967 18295 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.189:42113
I20260928 02:48:07.596998 18362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.189:42113 every 8 connection(s)
I20260928 02:48:07.597344 18295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/data/info.pb
I20260928 02:48:07.598277 18363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:07.599800 18363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:07.601233 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 18295
I20260928 02:48:07.601315 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-1/wal/instance
I20260928 02:48:07.601505 28086 external_mini_cluster.cc:1428] Setting key 3193aa4cff04a6a89661574503d1f205
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.188@KRBTEST.COM; defaulting to no policy
I20260928 02:48:07.609194 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.600441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.27.109.190:40201)
0928 02:48:07.600528 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.600530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.600715 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:48:07.601393 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.601394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.601410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.601465 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.602085 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.602708 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.603704 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.604069 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.604139 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.606102 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.606104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.606104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.606113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.607804 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.608182 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.608183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.608184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.608249 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.608481 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.608482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.608483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.608626 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.608705 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.608984 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.609077 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:48:07.609299 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.600389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40201 (local address 127.0.0.1:33000)
0928 02:48:07.600595 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.600635 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.601051 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.601567 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.601572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.601810 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.601986 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.601990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.602831 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.602832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.603621 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.603624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.604158 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.604159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.604258 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.604739 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.604747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.606009 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.607886 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.607892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.607894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.608103 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.608318 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.608319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.608319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.608411 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.608721 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.608727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.608845 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.608936 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.609121 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260928 02:48:07.609843 18363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.188@KRBTEST.COM" created.
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:07.617573 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.610084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59694)
0928 02:48:07.610138 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.610149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.610189 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.610410 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.610411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.610576 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.610623 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.610624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.611639 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.611639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.612435 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.612438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.613019 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.613020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.613060 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.613489 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.613493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.614669 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.616414 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.616415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.616416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.616642 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.616841 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.616842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.616842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.616911 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.617180 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.617181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.617227 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.617300 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.617403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:48:07.617576 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.610145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.27.109.189:42113)
0928 02:48:07.610278 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.610282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.610307 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:48:07.610324 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.610326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.610335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.610395 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.610673 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.611528 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.612553 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.612866 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.612971 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.614750 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.614758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.614758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.614766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.616314 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.616729 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.616731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.616731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.616780 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.616977 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.616978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.616978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.617093 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.617191 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.617331 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.617448 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260928 02:48:07.618234 18363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:07.618579 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.618441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45486)
0928 02:48:07.618501 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.618548 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:48:07.618716 18313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
I20260928 02:48:07.619161 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.619053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45488)
0928 02:48:07.619103 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.619133 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 02:48:07.619346 18363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:36235: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
Entry for principal kudu/127.27.109.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab.
I20260928 02:48:07.626889 28086 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpfCjkR/build/release/bin/kudu
/tmp/dist-test-taskpfCjkR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:36235
--webserver_interface=127.27.109.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.148:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:48:07.650573 18288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } attempt: 3
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:07.657125 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.650918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.27.109.189:42113)
0928 02:48:07.650974 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.650977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.650987 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:48:07.651034 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.651035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.651045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.651116 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.651434 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.651909 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.652614 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.652849 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.652936 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.654727 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.654729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.654729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.654736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.656046 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.656389 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.656390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.656390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.656446 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.656639 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.656640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.656640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.656744 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.656820 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.656967 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.657035 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:48:07.657128 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.650854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59710)
0928 02:48:07.650913 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.650924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.650977 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.651135 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.651136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.651279 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.651344 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.651345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.651997 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.651998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.652530 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.652531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.652936 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.652937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.653022 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.653479 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.653484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.654644 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.656146 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.656147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.656148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.656316 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.656512 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.656513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.656513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.656568 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.656812 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.656813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.656856 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.656942 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.657034 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:48:07.657639 18288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:48:07.657977 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.657882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45504)
0928 02:48:07.657928 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.657954 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:48:07.658260 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.658172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45510)
0928 02:48:07.658212 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.658234 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:48:07.658351 18288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:36235: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
I20260928 02:48:07.668548 18363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } attempt: 1
I20260928 02:48:07.668853 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.668752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45512)
0928 02:48:07.668801 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.668825 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:48:07.669137 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.669043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45524)
0928 02:48:07.669094 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.669117 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:48:07.669226 18363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:36235: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
I20260928 02:48:07.689491 18288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } attempt: 1
I20260928 02:48:07.689857 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.689735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45528)
0928 02:48:07.689795 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.689829 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:48:07.690186 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.690095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45534)
0928 02:48:07.690138 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.690163 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:48:07.690276 18288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:36235: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
W20260928 02:48:07.705777 18370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:48:07.705992 18370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:48:07.706032 18370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:48:07.706066 18370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:48:07.707247 18370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:48:07.707309 18370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:48:07.707335 18370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:48:07.707368 18370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:48:07.707391 18370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:48:07.708786 18370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.148:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.190:40201,127.27.109.189:42113,127.27.109.188:36235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/krb5kdc/kudu_127.27.109.188.keytab
--principal=kudu/127.27.109.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.188:36235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:48:07.709043 18370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:48:07.709277 18370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:48:07.714020 18375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:07.714008 18376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:48:07.714007 18378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:48:07.714353 18370 server_base.cc:1034] running on GCE node
I20260928 02:48:07.714591 18370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:48:07.714865 18370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:48:07.716022 18370 hybrid_clock.cc:648] HybridClock initialized: now 1790563687716017 us; error 34 us; skew 500 ppm
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:48:07.718367 18370 init.cc:377] Logged in from keytab as kudu/127.27.109.188@KRBTEST.COM (short username kudu)
I20260928 02:48:07.718791 18370 webserver.cc:460] Webserver started at http://127.27.109.188:44691/ using document root <none> and password file <none>
I20260928 02:48:07.719000 18370 fs_manager.cc:359] Metadata directory not provided
I20260928 02:48:07.719065 18370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:48:07.719224 18370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:48:07.720191 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data/instance:
uuid: "62a1e567289f4db89a1da185357b0c14"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "22ac8393006b033fdf7620fef4ffeb7d"
server_key_iv: "bedd7edec635c4f82b656635da4cc994"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.720530 18370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal/instance:
uuid: "62a1e567289f4db89a1da185357b0c14"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "22ac8393006b033fdf7620fef4ffeb7d"
server_key_iv: "bedd7edec635c4f82b656635da4cc994"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.721745 18370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:48:07.722397 18385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:48:07.722566 18370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:48:07.722668 18370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data,/tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal
uuid: "62a1e567289f4db89a1da185357b0c14"
format_stamp: "Formatted at 2026-09-28 02:48:07 on dist-test-slave-b5p5"
server_key: "22ac8393006b033fdf7620fef4ffeb7d"
server_key_iv: "bedd7edec635c4f82b656635da4cc994"
server_key_version: "encryptionkey@0"
I20260928 02:48:07.722746 18370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:48:07.735498 18370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:48:07.737349 18363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } attempt: 2
I20260928 02:48:07.737682 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.737572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45546)
0928 02:48:07.737626 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.737658 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:48:07.737699 18370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:48:07.737847 18370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:48:07.737990 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.737897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45562)
0928 02:48:07.737943 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.737965 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:48:07.738068 18363 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.188:36235: Network error: Client connection negotiation failed: client connection to 127.27.109.188:36235: connect: Connection refused (error 111)
I20260928 02:48:07.741603 18370 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.188:36235
I20260928 02:48:07.741631 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.188:36235 every 8 connection(s)
I20260928 02:48:07.741956 18370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/data/info.pb
I20260928 02:48:07.742782 18438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:48:07.744153 18438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:07.750869 28086 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpfCjkR/build/release/bin/kudu as pid 18370
I20260928 02:48:07.750983 28086 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpfCjkR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563399140758-28086-0/minicluster-data/master-2/wal/instance
I20260928 02:48:07.751190 28086 external_mini_cluster.cc:1428] Setting key 0886a9b92a412915f55c0ad4ded5c157
I20260928 02:48:07.752794 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.744803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.27.109.190:40201)
0928 02:48:07.744862 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.744864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.745031 (+ 167us) server_negotiation.cc:407] Connection header received
0928 02:48:07.745458 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.745459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.745467 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.745520 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.746035 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.746555 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.747516 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.747700 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.747762 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.749871 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.749873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.749873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.749879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.751398 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.751855 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.751857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.751858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.751922 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.752177 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.752178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.752178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.752289 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.752365 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.752620 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.752698 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:48:07.752952 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.744777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.190:40201 (local address 127.0.0.1:33016)
0928 02:48:07.744912 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.744949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.745379 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.745568 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.745571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.745791 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.745969 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.745973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.746636 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.746637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.747439 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.747441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.747848 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.747850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.747963 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.748494 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.748502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.749785 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.751480 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.751487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.751489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.751772 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.751996 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.751997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.751997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.752103 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.752356 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.752362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.752492 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.752597 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.752736 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260928 02:48:07.753489 18438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:07.760425 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.753828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.27.109.189:42113)
0928 02:48:07.753879 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.753881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.753890 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:48:07.753914 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.753914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.753924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.753972 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.754216 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.754754 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.755588 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.755811 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.755885 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.757694 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.757695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.757696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.757721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.759278 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759625 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.759625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.759626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.759700 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759909 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.759910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.759910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.760042 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.760121 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.760255 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.760336 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:48:07.760439 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.753766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.189:42113 (local address 127.0.0.1:59726)
0928 02:48:07.753811 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.753821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.753858 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.753972 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.753973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.754102 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.754142 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.754143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.754841 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.754842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.755489 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.755491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.755882 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.755884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.755944 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.756325 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.756330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.757574 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.759351 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.759353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.759556 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.759776 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.759778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.759839 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.760111 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.760111 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.760142 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.760217 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.760355 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":61}
I20260928 02:48:07.760553 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.751554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.27.109.188:36235)
0928 02:48:07.751732 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.751736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.751901 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:48:07.751960 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.751962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.751976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.752069 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.752667 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.753701 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.754496 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.754795 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.754874 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.757190 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.757197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.757197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.757203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.759073 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759430 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.759432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.759433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.759484 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.759688 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.759689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.759689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.759860 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.760022 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.760211 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.760303 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260928 02:48:07.760998 18438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.188@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.190@KRBTEST.COM
I20260928 02:48:07.765410 18288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } attempt: 2
I20260928 02:48:07.767948 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.761245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45580)
0928 02:48:07.761333 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.761343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.761390 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.761496 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.761496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.761616 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.761639 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.761640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.762366 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.762367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.762934 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.762936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.763383 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.763384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.763433 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.763922 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.763926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.764979 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.766867 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.766867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.766868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.767093 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.767310 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.767311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.767312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.767376 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.767634 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.767641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.767685 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.767758 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.767832 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:48:07.768419 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.761275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.27.109.188:36235)
0928 02:48:07.761306 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.761308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.761377 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:48:07.761436 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.761437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.761445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.761493 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.761710 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.762296 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.763007 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.763321 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.763368 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.765058 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.765059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.765060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.765065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.766787 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.767165 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.767166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.767167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.767224 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.767447 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.767448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.767448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.767561 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.767621 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.768255 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.768337 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:48:07.768929 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.761293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.27.109.190:40201)
0928 02:48:07.761330 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.761331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.761342 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:48:07.761403 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.761404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.761413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.761463 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.762011 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.762544 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.763235 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.763460 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.763530 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.765837 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.765839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.765839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.765846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.767565 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.767962 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.767963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.767964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.768021 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.768244 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.768245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.768246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.768369 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.768513 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.768747 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.768820 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.190@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:48:07.769879 18438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: Bootstrap starting.
I20260928 02:48:07.770438 18438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:07.770857 18438 log.cc:824] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:07.771700 18438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: No bootstrap required, opened a new log
I20260928 02:48:07.773805 18438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.773999 18438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a1e567289f4db89a1da185357b0c14, State: Initialized, Role: FOLLOWER
I20260928 02:48:07.774112 18438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.774336 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.765615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45586)
0928 02:48:07.765657 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.765666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.765701 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.765951 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.765952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.766110 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.766147 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.766148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.767021 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.767022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.767804 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.767808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.768348 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.768349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.768410 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.768849 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.768853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.770219 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.773164 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.773165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.773166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.773391 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.773619 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.773621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.773621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.773688 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.773990 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.773990 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.774031 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.774109 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.774222 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:48:07.774351 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.765671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.27.109.188:36235)
0928 02:48:07.765813 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.765815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.765822 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:48:07.765840 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.765842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.765851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.766216 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.766266 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.766927 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.767893 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.768252 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.768342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.770318 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.770320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.770320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.770326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.773047 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.773475 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.773477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.773477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.773541 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.773765 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.773766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.773767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.773914 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.773985 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.774165 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.774247 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260928 02:48:07.774394 18438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.189@KRBTEST.COM
I20260928 02:48:07.776228 18288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: Bootstrap starting.
I20260928 02:48:07.776787 18288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:07.777074 18288 log.cc:824] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:07.777351 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.777441 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.777907 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.769779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.27.109.189:42113)
0928 02:48:07.769815 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.769817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.769856 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:48:07.769924 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.769925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.769934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.769980 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.770894 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.771549 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.772429 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.772601 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.772680 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.775276 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.775277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.775277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.775284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.776709 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.777062 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.777063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.777064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.777114 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.777336 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.777338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.777338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.777479 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.777629 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.777754 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.777820 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:07.778105 18457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:07.778151 18457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:07.778203 18288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: No bootstrap required, opened a new log
I20260928 02:48:07.779137 28086 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:48:07.780015 18262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3785239362cf405cb1c6b25e3931dc6b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33022
I20260928 02:48:07.780258 18288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.780388 18288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3785239362cf405cb1c6b25e3931dc6b, State: Initialized, Role: FOLLOWER
I20260928 02:48:07.780462 18288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.780764 18288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:07.780795 18262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:48:07.780916 18262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:07.778782 18370 master_runner.cc:186] Error getting master registration for 127.27.109.190:40201: OK, instance_id { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" instance_seqno: 1790563687424614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:48:07.780992 18370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:07.781636 18262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.781827 18262 leader_election.cc:288] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [CANDIDATE]: Term 1 election: Requested vote from peers 7e1cd932465f4cebb1a1d3e92edef78a (127.27.109.189:42113), 62a1e567289f4db89a1da185357b0c14 (127.27.109.188:36235)
I20260928 02:48:07.782091 18412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3785239362cf405cb1c6b25e3931dc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62a1e567289f4db89a1da185357b0c14"
I20260928 02:48:07.782212 18412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:48:07.783111 18470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:07.783164 18470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:07.783215 18412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3785239362cf405cb1c6b25e3931dc6b in term 1.
I20260928 02:48:07.783414 18238 leader_election.cc:302] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3785239362cf405cb1c6b25e3931dc6b, 62a1e567289f4db89a1da185357b0c14; no voters:
I20260928 02:48:07.783607 18472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:48:07.783855 18472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [term 1 LEADER]: Becoming Leader. State: Replica: 3785239362cf405cb1c6b25e3931dc6b, State: Running, Role: LEADER
I20260928 02:48:07.780755 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.784054 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.783999 18472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.784500 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3785239362cf405cb1c6b25e3931dc6b. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.784590 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:07.783640 18220 master_runner.cc:186] Error getting master registration for 127.27.109.189:42113: OK, instance_id { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" instance_seqno: 1790563687595758 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:48:07.786237 18220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:48:07.788251 18479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:48:07.788430 18479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:48:07.794842 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.786734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.27.109.190:40201)
0928 02:48:07.786770 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.786772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.786903 (+ 131us) server_negotiation.cc:407] Connection header received
0928 02:48:07.786961 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.786961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.786984 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.787031 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.787763 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.788500 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.789556 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.789831 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.789869 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.791253 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.791254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.791254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.791261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.792829 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.793149 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.793150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.793150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.793208 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.794214 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.794215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.794215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.794357 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.794480 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.794679 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.794749 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:48:07.795322 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.787264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.27.109.189:42113)
0928 02:48:07.787306 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.787308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.787617 (+ 309us) server_negotiation.cc:407] Connection header received
0928 02:48:07.787771 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.787772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.787782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.787811 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.788858 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.789547 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.790566 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.790917 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.791000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.792332 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.792334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.792335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.792345 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.794024 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.794403 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.794404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.794404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.794481 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.794658 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.794659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.794659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.794799 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.794884 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.795117 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.795200 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:48:07.802163 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.787616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.27.109.188:36235)
0928 02:48:07.787648 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.787650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.795590 (+ 7940us) server_negotiation.cc:407] Connection header received
0928 02:48:07.795649 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.795650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.795660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.795714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.796231 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.796957 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.797613 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.797832 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.797904 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.799203 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.799205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.799205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.799211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.800898 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.801290 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.801291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.801291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.801352 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.801590 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.801591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.801591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.801728 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.801804 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.801998 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.802074 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:48:07.815367 18412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [term 1 FOLLOWER]: Refusing update from remote peer 3785239362cf405cb1c6b25e3931dc6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:07.815788 18476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:48:07.817273 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3785239362cf405cb1c6b25e3931dc6b. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.817358 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.818576 18483 mvcc.cc:205] Tried to move back new op lower bound from 7334148865289138176 to 7334148865164419072. Current Snapshot: MvccSnapshot[applied={T|T < 7334148865289138176}]
I20260928 02:48:07.818841 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.818922 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:07.819413 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.819494 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.819794 18479 catalog_manager.cc:1347] Generated new cluster ID: 43e6c15080c04d5ca693ce17d3b3f47a
I20260928 02:48:07.819837 18479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:48:07.825731 18484 mvcc.cc:205] Tried to move back new op lower bound from 7334148865289138176 to 7334148865164419072. Current Snapshot: MvccSnapshot[applied={T|T < 7334148865289138176}]
I20260928 02:48:07.829187 18363 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } attempt: 3
Sep 28 02:48:07 dist-test-slave-b5p5 krb5kdc[18207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563687, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.189@KRBTEST.COM for kudu/127.27.109.188@KRBTEST.COM
I20260928 02:48:07.837507 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.829404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.188:36235 (local address 127.0.0.1:45600)
0928 02:48:07.829475 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:48:07.829690 (+ 215us) client_negotiation.cc:174] Beginning negotiation
0928 02:48:07.829734 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:48:07.829938 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:48:07.829938 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:48:07.830096 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:48:07.830138 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.830139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.831059 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.831061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.831863 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:48:07.831865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.832382 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.832383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:48:07.832457 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.832869 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:48:07.832873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:48:07.834367 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:48:07.836157 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.836158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.836159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.836382 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.836624 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:48:07.836625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:48:07.836625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:48:07.836696 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:48:07.837202 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:48:07.837203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:48:07.837265 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:48:07.837327 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:48:07.837410 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:48:07.837982 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:07.829639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.27.109.188:36235)
0928 02:48:07.829670 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.829672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.829784 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:48:07.829802 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.829803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.829812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.829853 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.830212 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.830969 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.831956 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:07.832256 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:07.832313 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:07.834493 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:48:07.834495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:48:07.834495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:48:07.834501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:48:07.836050 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.836474 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.836475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.836476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.836541 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:48:07.836961 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:48:07.836962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:48:07.836962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:48:07.837100 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:48:07.837158 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:07.837826 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 02:48:07.837899 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:48:07.839663 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: Bootstrap starting.
I20260928 02:48:07.840353 18363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: Neither blocks nor log segments found. Creating new log.
I20260928 02:48:07.840624 18363 log.cc:824] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:48:07.841581 18363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: No bootstrap required, opened a new log
I20260928 02:48:07.843511 18363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.843636 18363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1cd932465f4cebb1a1d3e92edef78a, State: Initialized, Role: FOLLOWER
I20260928 02:48:07.843703 18363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } }
I20260928 02:48:07.843969 18363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:48:07.845220 18336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:48:07.845981 18336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [term 1 FOLLOWER]: Refusing update from remote peer 3785239362cf405cb1c6b25e3931dc6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:48:07.846138 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3785239362cf405cb1c6b25e3931dc6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e1cd932465f4cebb1a1d3e92edef78a"
I20260928 02:48:07.846845 18337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3785239362cf405cb1c6b25e3931dc6b in term 1.
I20260928 02:48:07.847256 18476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:48:07.849988 18500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:48:07.850080 18500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:48:07.850492 18502 mvcc.cc:205] Tried to move back new op lower bound from 7334148865289138176 to 7334148865164419072. Current Snapshot: MvccSnapshot[applied={T|T < 7334148865289138176}]
I20260928 02:48:07.850769 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3785239362cf405cb1c6b25e3931dc6b. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.850922 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.851006 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.851099 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.854187 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.854295 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b [sys.catalog]: This master's current role is: LEADER
I20260928 02:48:07.855541 18479 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:48:07.848888 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3785239362cf405cb1c6b25e3931dc6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3785239362cf405cb1c6b25e3931dc6b" member_type: VOTER last_known_addr { host: "127.27.109.190" port: 40201 } } peers { permanent_uuid: "7e1cd932465f4cebb1a1d3e92edef78a" member_type: VOTER last_known_addr { host: "127.27.109.189" port: 42113 } } peers { permanent_uuid: "62a1e567289f4db89a1da185357b0c14" member_type: VOTER last_known_addr { host: "127.27.109.188" port: 36235 } } }
I20260928 02:48:07.855623 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:48:07.856336 18479 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:48:07.866345 18479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3785239362cf405cb1c6b25e3931dc6b: Generated new TSK 0
I20260928 02:48:07.866565 18479 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:48:07.899675 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.897995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.27.109.189:42113)
0928 02:48:07.898041 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.898043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.898993 (+ 950us) server_negotiation.cc:407] Connection header received
0928 02:48:07.899102 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.899103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.899115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.899162 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.899606 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59746: BlockingRecv error: recv got EOF from 127.0.0.1:59746 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:48:07.900458 18443 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.898365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.27.109.188:36235)
0928 02:48:07.898408 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.898410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.899943 (+ 1533us) server_negotiation.cc:407] Connection header received
0928 02:48:07.900040 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.900041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.900051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.900081 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.900401 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45602: BlockingRecv error: recv got EOF from 127.0.0.1:45602 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:48:07.901278 18291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.898717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.27.109.190:40201)
0928 02:48:07.898757 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.898758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.900655 (+ 1897us) server_negotiation.cc:407] Connection header received
0928 02:48:07.900802 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.900803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.900815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.900844 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.901217 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33040: BlockingRecv error: recv got EOF from 127.0.0.1:33040 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:48:07.907557 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.902480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.27.109.189:42113)
0928 02:48:07.902513 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.902515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.907014 (+ 4499us) server_negotiation.cc:407] Connection header received
0928 02:48:07.907155 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.907156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.907175 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.907205 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.907510 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59748: BlockingRecv error: recv got EOF from 127.0.0.1:59748 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:07.910538 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.909742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.27.109.189:42113)
0928 02:48:07.909784 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.909786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.910012 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:48:07.910105 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.910106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.910116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.910158 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.910494 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59758: BlockingRecv error: recv got EOF from 127.0.0.1:59758 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:48:07.914258 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.913553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.27.109.189:42113)
0928 02:48:07.913584 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.913586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.913798 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:48:07.913889 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.913890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.913899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.913926 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.914219 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59764: BlockingRecv error: recv got EOF from 127.0.0.1:59764 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 02:48:07.920228 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.919482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.27.109.189:42113)
0928 02:48:07.919512 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.919514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.919741 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:48:07.919829 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.919830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.919839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.919866 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.920173 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59778: BlockingRecv error: recv got EOF from 127.0.0.1:59778 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:48:07.929922 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.929219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.27.109.189:42113)
0928 02:48:07.929250 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.929252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.929483 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:48:07.929565 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.929566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.929575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.929602 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.929875 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59790: BlockingRecv error: recv got EOF from 127.0.0.1:59790 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:48:07.947751 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.946974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.27.109.189:42113)
0928 02:48:07.947017 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.947019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.947271 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:48:07.947362 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.947363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.947372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.947399 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.947704 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59800: BlockingRecv error: recv got EOF from 127.0.0.1:59800 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:48:07.981945 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:07.981001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.27.109.189:42113)
0928 02:48:07.981052 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:07.981055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:07.981255 (+ 200us) server_negotiation.cc:407] Connection header received
0928 02:48:07.981336 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:07.981337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:07.981350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:07.981407 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:07.981875 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59804: BlockingRecv error: recv got EOF from 127.0.0.1:59804 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 02:48:08.047875 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:08.047195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.27.109.189:42113)
0928 02:48:08.047265 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.047267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.047370 (+ 103us) server_negotiation.cc:407] Connection header received
0928 02:48:08.047417 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.047418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.047429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.047492 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:08.047804 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59814: BlockingRecv error: recv got EOF from 127.0.0.1:59814 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:48:08.177645 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:08.176997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.27.109.189:42113)
0928 02:48:08.177092 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.177094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.177151 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:48:08.177211 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.177212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.177222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.177276 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:08.177591 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59830: BlockingRecv error: recv got EOF from 127.0.0.1:59830 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 02:48:08.435555 18367 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:48:08.434868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.27.109.189:42113)
0928 02:48:08.434946 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.434948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.435056 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:48:08.435135 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.435137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.435147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.435212 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:48:08.435511 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59836: BlockingRecv error: recv got EOF from 127.0.0.1:59836 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:48:08.778581 18457 catalog_manager.cc:1259] Loaded cluster ID: 43e6c15080c04d5ca693ce17d3b3f47a
I20260928 02:48:08.778649 18457 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: loading cluster ID for follower catalog manager: success
I20260928 02:48:08.779457 18457 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: acquiring CA information for follower catalog manager: success
I20260928 02:48:08.779757 18457 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62a1e567289f4db89a1da185357b0c14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:08.850520 18500 catalog_manager.cc:1259] Loaded cluster ID: 43e6c15080c04d5ca693ce17d3b3f47a
I20260928 02:48:08.850586 18500 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: loading cluster ID for follower catalog manager: success
I20260928 02:48:08.851397 18500 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: acquiring CA information for follower catalog manager: success
I20260928 02:48:08.851701 18500 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e1cd932465f4cebb1a1d3e92edef78a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:48:08.951453 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:08.948679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.27.109.189:42113)
0928 02:48:08.948805 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.948807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.948875 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:48:08.948970 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.948971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.949030 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.949139 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:08.949319 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.949811 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.950621 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.950833 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.950912 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:08.951078 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:08.951195 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:08.951291 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:08.951332 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:48:08.951351 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260928 02:48:08.955400 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:08.953225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.27.109.188:36235)
0928 02:48:08.953322 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.953324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.953347 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:48:08.953388 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.953389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.953420 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.953477 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:08.953558 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.954010 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.954719 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.954916 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.954964 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:08.955071 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:08.955149 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:08.955196 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:08.955275 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:48:08.955292 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":53,"threads_started":1}
I20260928 02:48:08.959695 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:08.956871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.27.109.189:42113)
0928 02:48:08.956910 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.956912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.957050 (+ 138us) server_negotiation.cc:407] Connection header received
0928 02:48:08.957187 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.957188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.957198 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.957247 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:08.957345 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.958095 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.958967 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.959239 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.959309 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:08.959438 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:08.959470 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:08.959523 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:08.959580 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:48:08.959600 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:48:08.959964 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:08.957140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.27.109.188:36235)
0928 02:48:08.957174 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.957176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.957273 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:48:08.957382 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.957383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.957392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.957442 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:08.957574 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.958236 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.959207 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.959485 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.959543 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:08.959715 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:08.959743 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:08.959772 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:08.959889 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:48:08.959909 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:48:08.960079 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:48:08.956879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.27.109.190:40201)
0928 02:48:08.956971 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:48:08.956974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:48:08.957124 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:48:08.957172 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:48:08.957173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:48:08.957225 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:48:08.957301 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:48:08.957509 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.958009 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.959032 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:48:08.959351 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:48:08.959410 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:48:08.959722 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:48:08.959841 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:48:08.959897 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:48:08.959970 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 02:48:08.959995 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":48,"threads_started":1}
I20260928 02:48:08.961714 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 18220
I20260928 02:48:08.966832 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 18295
I20260928 02:48:08.971271 28086 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpfCjkR/build/release/bin/kudu with pid 18370
2026-09-28T02:48:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2794 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5586 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (289838 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS